HPCC-17112 Regression suite to check warnings

Add code to collect, check and report eclcc generated warnings.

Add elcccwarn file to loopif.ecl and xmlout2.ecl files as known issues.

Update README.rst file to explain how warning check works and
how to prepare .eclccwarn file.

Signed-off-by: HPCCSmoketest <hpccsmoketest@gmail.com>

testing/regress/README.rst

@@ -27,6 +27,7 @@ Result:
 |                       [--noversion]
 |                       [--runclass class[,class,...]]
 |                       [--excludeclass class[,class,...]]
+|                       [--handleEclccWarningFile]
 |                       {list,setup,run,query} ...
 | 
 |       HPCC Platform Regression suite
@@ -59,6 +60,8 @@ Result:
 |                                 run subclass(es) of the suite. Default value is 'all'
 |        --excludeclass class[,class,...], -e class[,class,...]
 |                                 exclude subclass(es) of the suite. Default value is 'none'
+|        --handleEclccWarningFile, -w
+|                                 Create/overwrite/delete ECLCC warning file.
 |
 
 Important!
@@ -821,3 +824,67 @@ If your HPCC System is configured to use LDAP authentication you should change v
 
 Alternatively, ensure that your test system has a user "regress" with password "regress" and appropriate rights to be able to run the suite.
 
+
+11. Handling ECLCC warnings:
+----------------------------
+There is a new feature of the Regression Test Engine: Eclcc warning check.
+
+With this feature, the engine checks the Eclcc compiler output (stderr stream) for every ECL test cases and looking for warnings.
+
+The possible events are:
+Test pass:
+    1. The test compiled without any warning. In this case the execution continuous as previously.
+    2. The test compiled with warnings, but the engine found ‘.eclccwarn’ file with all warnings. In this case the state is well known  and the test execution continuous as previously
+
+Test failing:
+    3. Suddenly the test compiled with one or more warnings. If this situation is new no eclccwarn file associated to that test case then the engine reports those new warnings as error and the test aborted.
+    4. The test compiled with warnings, the engine found .eclccwarn file, but there is some difference (warning(s) appear or disappear). In this case engine reports the difference between current compiler output and the state stored in .eclccwarn file. Further execution of test is  aborted
+    5. The test compiled without warnings, but the engine found .eclccwarn file. This means the warning(s) suddenly/unintentionally disappeared and the engine reports that changes and abort the test.
+
+For this checking the in events 2-5 the engine need an .eclccwarn file. To generate that file there is two ways:
+    1. Manually: 
+        a. In this case the ECL code should  compile with eclcc command like this:
+              eclcc  <ecl_file>.ecl  2> <ecl_file>.eclccwarn
+           with the stderr stream redirected into a file
+
+        b. Because the warning report contains the path to the ECL file and this path can be different from system to system and execution by execution (OBT, Smoketest, developer environment, etc.) all path should remove from the generated <ecl_file>.eclccwarn file. 
+
+        c. The edited <ecl_file>.eclccwarn must copy to the same place where the associated key file (<ecl_file>.xml) located.
+
+        d. Example:
+            i. Here is a simple ECL file with one line of code:
+                    '1'[1..2]
+               stored in ‘ecl/eclccwarning.ecl’ file.
+
+            ii. Execute it with:
+                    eclcc ecl/eclccwarning.ecl 2>eclccwarning.eclccwarn
+
+            iii. The content of the ‘eclccwarning.eclccwarn’ file is:
+                    ecl/eclccwarning.ecl(1,5): warning C2121: Invalid substring range: index 2 out of bound: 1..1
+                    0 error, 1 warning
+
+                So, the ‘.eclccwarn’ file contains the path ‘ecl/’ and in must remove:
+                    eclccwarning.ecl(1,5): warning C2121: Invalid substring range: index 2 out of bound: 1..1
+                    0 error, 1 warning
+                
+            iv. Copy the edited file into ecl/key/ directory and the next run of Regression Test Engine it will be used to check compiler warnings.
+           
+    2. Automated: (Warning!!! This is an easier but dangerous way!)
+        a. In this case the ECL code can run with Regression Test Engine like this:
+                ./ecl-test query –t <target_cluster> –w <ecl_file>.ecl  
+           The newly implemented –w or --handleEclccWarningFile parameter force the engine to create, rewrite or delete the <ecl_file>.eclccwarn file. Depend on the result of warning check.
+
+        b. This means
+            i. In event 3 a new warning file created.
+            
+            ii. In event 4 the existing warnings file overwritten by a new result
+                Warning! If appearing/disappearing of warning is not intentional, the previous warning state lost.
+                
+            iii. In event 5, all warnings disappeared the warning file is deleted.
+                Warning! It is same problem as II.
+                
+        c. Important! 
+           The –w or  --handleEclccWarningFile parameter working with query with wildcards and run mode and can cause to overwrite or remove all associated warning files.
+
+Last comment: the warning file is part of the (Regression) suite, so it must be handled same way as the ECL test code and the test related key file. 
+

testing/regress/ecl-test

@@ -165,6 +165,8 @@ class RegressMain:
                                 nargs=1,  default = ['all'],   metavar="class[,class,...]")
         executionParser.add_argument('--excludeclass', '-e', help="exclude subclass(es) of the suite. Default value is 'none'",
                                 nargs=1,  default = ['none'],   metavar="class[,class,...]")
+        executionParser.add_argument('--handleEclccWarningFile', '-w', help="Create/overwrite/delete ECLCC warning file.",
+                                action='store_true')
 
         parser = argparse.ArgumentParser(prog=prog, description=description,  parents=[helperParser, commonParser,  executionParser])
 

testing/regress/ecl/key/loopif.eclccwarn

@@ -0,0 +1,2 @@
+loopif.ecl(27,68): warning C2168: Field 'i' in TABLE does not appear to be properly defined by grouping conditions
+0 error, 1 warning

testing/regress/ecl/key/xmlout2.eclccwarn

@@ -0,0 +1,2 @@
+xmlout2.ecl(66,38): warning C1023: Condition is always false
+0 error, 1 warning

testing/regress/hpcc/common/shell.py

@@ -58,4 +58,4 @@ class Shell:
             exception.output = err_msg
             logging.debug("exception.output:'%s'",  err_msg)
             raise Error('1001', err=str(err_msg))
-        return stdout
+        return stdout, stderr

testing/regress/hpcc/regression/regress.py

@@ -504,35 +504,43 @@ class Regression:
         res = 0
         wuid = None
         if ECLCC().makeArchive(query):
-            eclCmd = ECLcmd()
-            try:
-                if publish:
-                    res = eclCmd.runCmd("publish", cluster, query, report[0],
-                                      server=self.config.espIp,
-                                      username=self.config.username,
-                                      password=self.config.password, 
-                                      retryCount=self.config.maxAttemptCount)
-                else:
-                    res = eclCmd.runCmd("run", cluster, query, report[0],
-                                      server=self.config.espIp,
-                                      username=self.config.username,
-                                      password=self.config.password, 
-                                      retryCount=self.config.maxAttemptCount)
-            except Error as e:
-                logging.debug("Exception raised:'%s'"  % ( str(e)),  extra={'taskId':cnt})
+            if query.isEclccWarningChanged():
+                logging.debug("Should check Eclcc Warning:'%s'"  % (query.getEclccWarning()),  extra={'taskId':cnt})
                 res = False
                 wuid = 'Not found'
                 query.setWuid(wuid)
-                query.diff = query.getBaseEcl()+"\n\t"+str(e)
+                query.diff = query.getEclccWarningChanges()
                 report[0].addResult(query)
-                pass
-            except:
-                logging.error("Unexpected error:'%s'" %( sys.exc_info()[0]) ,  extra={'taskId':cnt})
-
-            wuid = query.getWuid()
-            logging.debug("CMD result: '%s', wuid:'%s'"  % ( res,  wuid),  extra={'taskId':cnt})
-            if wuid == 'Not found':
-                res = False
+            else:
+                eclCmd = ECLcmd()
+                try:
+                    if publish:
+                        res = eclCmd.runCmd("publish", cluster, query, report[0],
+                                          server=self.config.espIp,
+                                          username=self.config.username,
+                                          password=self.config.password,
+                                          retryCount=self.config.maxAttemptCount)
+                    else:
+                        res = eclCmd.runCmd("run", cluster, query, report[0],
+                                          server=self.config.espIp,
+                                          username=self.config.username,
+                                          password=self.config.password,
+                                          retryCount=self.config.maxAttemptCount)
+                except Error as e:
+                    logging.debug("Exception raised:'%s'"  % ( str(e)),  extra={'taskId':cnt})
+                    res = False
+                    wuid = 'Not found'
+                    query.setWuid(wuid)
+                    query.diff = query.getBaseEcl()+"\n\t"+str(e)
+                    report[0].addResult(query)
+                    pass
+                except:
+                    logging.error("Unexpected error:'%s'" %( sys.exc_info()[0]) ,  extra={'taskId':cnt})
+
+                wuid = query.getWuid()
+                logging.debug("CMD result: '%s', wuid:'%s'"  % ( res,  wuid),  extra={'taskId':cnt})
+                if wuid == 'Not found':
+                    res = False
         else:
             res = False
             report[0].addResult(query)

testing/regress/hpcc/util/ecl/cc.py

@@ -40,7 +40,7 @@ class ECLCC(Shell):
         except Error as err:
             logging.debug("getArchive exception:'%s'",  repr(err))
             self.makeArchiveError = str(err)
-            return repr(err)
+            return (repr(err), repr( err))
 
     def makeArchive(self, ecl):
         self.addIncludePath(ecl.dir_inc)
@@ -51,7 +51,7 @@ class ECLCC(Shell):
             os.mkdir(dirname)
         if os.path.isfile(filename):
             os.unlink(filename)
-        result = self.getArchive(ecl)
+        result, stderr = self.getArchive(ecl)
 
         if result.startswith( 'Error()'):
             retVal = False
@@ -73,6 +73,9 @@ class ECLCC(Shell):
                 ecl.diff += repr(self.makeArchiveError)
             self.makeArchiveError=''
         else:
+            logging.debug("%3d. makeArchive (stderr:'%s')", ecl.getTaskId(), stderr )
+            if 'arning' in stderr:
+                ecl.setEclccWarning(stderr)
             logging.debug("%3d. makeArchive (filename:'%s')", ecl.getTaskId(), filename )
             FILE = open(filename, "w")
             FILE.write(result)

testing/regress/hpcc/util/ecl/command.py

@@ -88,8 +88,9 @@ class ECLcmd(Shell):
         results=''
         try:
             #print "runCmd:", args
-            results = self.__ECLcmd()(*args)
+            results, stderr = self.__ECLcmd()(*args)
             logging.debug("%3d. results:'%s'", eclfile.getTaskId(),  results)
+            logging.debug("%3d. stderr :'%s'", eclfile.getTaskId(),  stderr)
             data = '\n'.join(line for line in
                              results.split('\n') if line) + "\n"
             ret = data.split('\n')

testing/regress/hpcc/util/ecl/file.py

@@ -82,6 +82,8 @@ class ECLFile:
         self.versionId=0
         self.timeout = 0
         self.args = args
+        self.eclccWarning = ''
+        self.eclccWarningChanges = ''
 
         #If there is a --publish CL parameter then force publish this ECL file
         self.forcePublish=False
@@ -527,3 +529,73 @@ class ECLFile:
     def getDParameters(self):
         logging.debug("%3d. getDParameters (ecl:'%s', D parameters are:'%s')", self.taskId,  self.ecl, self.paramD)
         return self.paramD
+
+    def setEclccWarning(self,  warning):
+        logging.debug("%3d. setEclccWarning (ecl:'%s', warning(s) is:'%s')", self.taskId,  self.ecl, warning)
+        self.eclccWarning = warning.strip().replace(self.dir_ec+'/','').split('\n')
+
+    def getEclccWarning(self):
+        logging.debug("%3d. getEclccWarning (ecl:'%s', warning(s) is:'%s')", self.taskId,  self.ecl, self.eclccWarning)
+        return self.eclccWarning
+
+    def isEclccWarningChanged(self):
+        retVal = False
+        expectedKeyPath = self.getExpected()
+        expectedKeyPath = expectedKeyPath.replace('.xml', '.eclccwarn')
+        logging.debug("%3d. EXP: " + expectedKeyPath,  self.taskId )
+        eclccKeyContent = []
+        if os.path.isfile(expectedKeyPath):
+            # put warning file content into self.eclccWarningChanges
+            eclccKeyContent = open(expectedKeyPath, 'r').readlines()
+            eclccKeyContent = [x.strip() for x in eclccKeyContent]
+            logging.debug("%3d. eclccKeyContent: " + "\n".join(eclccKeyContent),  self.taskId )
+        elif '' != self.eclccWarning:
+            logging.debug("%3d. Eclcc warning file '%s' doesn't exist." % (self.taskId, expectedKeyPath))
+            eclccKeyContent = []
+            if self.args.handleEclccWarningFile:
+                logging.debug("%3d. Create '%s' eclcc warning file." %(self.taskId, expectedKeyPath ))
+                open(expectedKeyPath, 'w').write("\n".join(self.eclccWarning))
+                pass
+        try:
+            diffLines = ''
+            d = list(difflib.unified_diff(eclccKeyContent, self.eclccWarning, fromfile=expectedKeyPath, tofile="eclcc warning",  lineterm = ""))
+            diffLines = "\n".join(d)
+
+            logging.debug("%3d. diffLines: " + diffLines,  self.taskId )
+            if len(diffLines) > 0:
+                self.eclccWarningChanges += ("%3d. Test: %s\n") % (self.taskId,  self.getBaseEclRealName())
+                self.eclccWarningChanges += "\tEclcc generated warning changed\n"
+                logging.debug( "type(diffLines) is %s: ",  repr(type(diffLines)), extra={'taskId':self.taskId})
+                if type(diffLines) == type(u' '):
+                    diffLines = unicodedata.normalize('NFKD', diffLines).encode('ascii','ignore').replace('\'','').replace('\\u', '\\\\u')
+                    diffLines = str(diffLines)
+                else:
+                    diffLines = str(diffLines)
+                self.eclccWarningChanges += str(diffLines)
+                retVal = True
+                if self.args.handleEclccWarningFile:
+                    if 0 < len(self.eclccWarning):
+                        logging.debug("%3d. Overwrite '%s' with current eclcc warning!" %(self.taskId, expectedKeyPath ))
+                        open(expectedKeyPath, 'w').write("\n".join(self.eclccWarning))
+                    else:
+                        logging.debug("%3d. Remove '%s' with current eclcc warning!" %(self.taskId, expectedKeyPath ))
+                        os.unlink(expectedKeyPath)
+                    pass
+            logging.debug("%3d. self.diff: '" + self.eclccWarningChanges +"'",  self.taskId )
+        except Exception as e:
+            logging.debug( e, extra={'taskId':self.taskId})
+            logging.debug("%s",  traceback.format_exc().replace("\n","\n\t\t"),  extra={'taskId':self.taskId} )
+            logging.debug("EXP: %s",  eclccKeyContent,  extra={'taskId':self.taskId})
+            logging.debug("REC: %s",  self.eclccWarning,  extra={'taskId':self.taskId})
+            retVal = True
+        finally:
+            if self.eclccWarningChanges == '':
+                retVal = False
+            else:
+                retVal = True
+
+        return retVal
+
+    def getEclccWarningChanges(self):
+        # return with self.eclccWarningChanges
+        return self.eclccWarningChanges+"\n"

testing/regress/hpcc/util/util.py

@@ -106,7 +106,7 @@ def queryWuid(jobname,  taskId):
     args.append('--server=' + gConfig.espIp)
     args.append('--username=' + gConfig.username)
     args.append('--password=' + gConfig.password)
-    res = shell.command(cmd, *defaults)(*args)
+    res, stderr = shell.command(cmd, *defaults)(*args)
     logging.debug("%3d. queryWuid(%s, cmd :'%s') result is: '%s'",  taskId,  jobname, cmd,  res)
     wuid = "Not found"
     state = 'N/A'