Merge branch 'candidate-5.2.0'

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>

Conflicts:
	common/workunit/workunit.hpp

charm/README

@@ -0,0 +1,80 @@
+HPCC Juju Charm Development Guide
+
+Table of Contents
+-----------------
+Introduction
+How to get source code
+How to open a bug
+How to work on a bug or new feature code
+
+
+Introduction
+------------
+HPCC Juju Charm helps users deploy HPCC Platform on various cloud environment.
+It can be deployed either directly from charm store or on local system image.
+It currently supports deploying HPCC to Ubuntu 12.04 amd64 (precise) and 14.04 amd64 (trusty).
+
+Development system is also on Ubuntu precise or trusty for easily test with local provider.
+
+There are not many differences between precise and trusty codes except HPCC images, checksum
+and README items.
+
+HPCC Juju Charm for precise:  https://jujucharms.com/hpcc/precise/
+HPCC Juju Charm for trusty:  https://jujucharms.com/hpcc/trusty/
+
+Juju charm Documentation: https://juju.ubuntu.com/docs/
+
+
+How to get source code
+----------------------
+1. Install charm-tools
+   sudo add-apt-repository ppa:juju/stable
+   sudo apt-get update
+   sudo apt-get install charm-tools
+   sudo apt-get bzr
+
+2. Get HPCC Juju Charm source for Ubuntu precise:
+   mkdir precise
+
+   download Download.zip from  https://jujucharms.com/hpcc/precise
+   or
+   charm get cs:precise/hpcc
+   or
+   bzr branch ip:~charmers/charms/precise/hpcc/trunk hpcc
+
+3. Get HPCC Juju Charm source for Ubuntu trusty:
+   mkdir trusty
+   download Download.zip from  https://jujucharms.com/hpcc/trusty
+   or
+   charm get cs:trusty/hpcc
+   or
+   bzr branch ip:~charmers/charms/trusty/hpcc/trunk hpcc
+
+How to open a bug
+----------------------------------------
+1. Create launchpad id if haven't at: https://launchpad.net
+2. https://bugs.launchpad.net/juju-core/+filebug
+
+
+How to work on a bug or new feature code
+----------------------------------------
+For development (fix bugs or add new features) you should checkout the source code
+from launchpad with bzr command as documented in "How to get source code" section.
+
+After you get the source code you can work with the code:
+1. Make your changes
+2. bzr commit -m "<comment>"
+3  bzr push lp:~<your-launchpad-username>/charms/<precise|trusty>/hpcc/<branch name>
+   This is where you could create a merge proposal with the hpcc charm
+
+4. If you are or  want to be the  owner of the bug
+   Click on "Link to related branch" in
+     https://bugs.launchpad.net/charms/+source/hpcc/+bug/<bug number>
+5. Assign the bug to yourself
+6. Change the status to "Fix Committed"
+7. That should enter your bug in the review queue. Juju Charm Development team will
+   assign a tester/developer to review it.
+
+
+Reference https://juju.ubuntu.com/docs/authors-charm-store.html#submitting-a-fix-to-an-existing-charm
+for more information

charm/precise/hpcc/README.md

@@ -1,105 +0,0 @@
-# Overview
-
-[HPCC Systems,](http://HPCCSystems.com) an open source High Performance Computing Cluster, is a massive parallel-processing computing platform that solves Big Data problems. HPCC Systems is an enterprise-proven platform for manipulating, transforming, querying, and data warehousing Big Data. Built by LexisNexis, the HPCC platform has helped it grow to a $1.5 billion information solutions company.
-
-The HPCC Systems architecture incorporates a data query engine (called Thor) and a data delivery engine (called Roxie), as well as common middleware support components, an external communications layer, client interfaces which provide both end-user services and system management tools, and auxiliary components to support monitoring and to facilitate loading and storing of file system data from external sources.
-
-An HPCC environment can include only Thor clusters, or both Thor and Roxie clusters. The HPCC Juju charm creates a cluster which contains both, but you can customize it after deployment.
-
-See [How it Works](http://www.hpccsystems.com/Why-HPCC/How-it-works)  for more details. 
-
-See [System Requirements](http://hpccsystems.com/download/docs/system-requirements) for  hardware details. 
-
-The HPCC Juju Charm encapsulates best practice configurations for the HPCC  Systems Platform.  You can use a Juju Charm to stand up an HPCC Platform on:
-
-- Local Provider (LXC)
-
-- Amazon Web Services Cloud
-
-
-# Usage
-
-## General Usage
-
-1. To deploy an HPCC Cluster:
-
-    `juju deploy hpcc <cluster_name>`
-
-    **For example:**
-
-        'juju deploy hpcc cluster1`
-
-1. To check the status , run
-        juju status
-
-        You also can log into the node to check if HPCC is properly installed.
-
-        `juju ssh cluster1/0`
-
-Once the service is deployed and running, you can find the address for the GUI by running juju status and looking for the public-address field for the juju-gui service
-
-1.  Once HPCC is properly installed, you can add more nodes using this command:
-
-        `juju add-unit <cluster_name> -n <#_of_nodes_to_add>`
-
-    **For example:**
-
-        `juju add-unit cluster1 -n 3`
-
-1. You can expose the HPCC cluster by running:
-
-       `juju expose <cluster_name>`
-
-Once the service is deployed, running, and exposed, you can find the address for the ECL Watch Web interface by running juju status and looking for the public-address field. Type that address plus :8010 for the port.
-
-For example, **nnn.nnn.nnn.nnn:8010**.
-
-If you have multiple nodes, the ECL Watch node will be the lowest IP address (first hpcc node listed).
-
-
-
-# Configuration
-
-When you use the `juju add-unit` command to add nodes, scripts are called automatically to provide a default configuration. 
-
-If you want to configure manually, set **auto-gen** to **0**, wait for all nodes to be in a "started" state, then call the **config_hpcc.sh**  script using the following parameters:
-
-`./config_hpcc.sh -thornodes <# of thor nodes> -roxienodes <# of roxie nodes> -supportnodes <# of support nodes> -slavespernode <#of thor slaves per node> 
-`
-
-Another useful script reports the URL for the ECL Watch node. Call the **get-url.sh** script to display the cluster configuration and the URL for the ECL Watch service.
-
-### ssh-keys ###
-The hpcc charm automatically generates a key pair  (*id\_rsa*  &  *id\_rsa.pub*) to configure nodes. 
-
-If you already have your own key pair and wish to use it, copy and paste their contents into the two variables (*ssh-key-public* and *ssh-key-private*) in the configuration file (config.yaml) or in the Juju canvas configuration settings.  
-
-### Verifying the checksum
-The charm uses an md5sum to verify the checksum of the HPCC platform  package before installing.  
-
-For this version of the charm, it is set to check the md5sum for the Community Edition Version 4.2.0-4 for Ubuntu 12.04. To verify a different version, edition, or OS version, change the value of the md5sum in the package-checksum variable in config.yaml. 
-
- 
-
-### AWS Cloud
-
-When deploying to Amazon Web Services Cloud, the charm automatically opens for external access, the following ports:
-
-- Port **8010** for ECLWatch access
-- Port **8002** for WsECL access.
-- Port **9876** for direct Roxie access
-- Port **8015** for Configuration Manager access.  
-
-### Next Steps ###
-
-After deploying and adding nodes, you can tweak various options to optimize your HPCC deployment to meet your needs.
- 
-See [HPCC Systems Web site](http://HPCCSystems.com) for more details.
-
-
-# HPCC Systems Contact Information
-
-[HPCC Systems Web site](http://HPCCSystems.com)
-
-For support, visit the HPCC Community Forums:
-[HPCC Community Forums](http://hpccsystems.com/bb/index.php?sid=0bda2dddb2ea50418357171d33b11e5f)

charm/precise/hpcc/bin/config_hpcc.sh

@@ -1,244 +0,0 @@
-#!/bin/bash
-
-function usage()
-{
-   cat <<EOF
-
-    Trigger HPCC envgen with charm configure changed hook
-    Usage:  $(basename $0) <options>
-    where
-       -name <service name> :
-          Juju charm service name. Then name can be provided at charm
-          deploy name. The default is the same as charm name.  This will
-          identify HPCC cluster.
-
-       -supportnodes <number of support nodes>:
-          Number of nodes to be used for non-Thor and non-Roxie components.
-          If not specified or specified as 0, thor and roxie nodes may
-          overlap with support nodes. If an invalid value is provided,
-          support nodes are treated to be 0
-
-       -roxienodes <number of roxie nodes>:
-          Number of nodes to be generated for roxie. If not specified or
-          specified as 0, no roxie nodes are generated
-
-       -thornodes <number of thor nodes>: Number of nodes to be generated
-          for thor slaves. A node for thor master is automatically added.
-          If not specified or specified as 0, no thor nodes are generated
-
-       -slavespernode <number of thor slaves per node>:
-          Number of thor nodes per slave.
-
-
-       -list:
-          List envgen optios only.
-
-       -updateonly :
-          Update config values only. Will not trigger envgen. This is
-          mainly for update values before add new nodes. So new nodes
-          relation changed hooks will generate desired environment.xml.
-
-    The value with "+" mean add to original number.
-    The value with "-" mean substract from original number.
-
-EOF
-}
-
-function get_service_information()
-{
-   _options=
-
-   [ -n "${service[service_name]}" ] && _options="-s ${service[service_name]}"
-
-   service_info=$(python ${ABS_CWD}/parse_status.py $_options)
-   service_list=( $service_info )
-   for item in "${service_list[@]}"
-   do
-       key=$(echo $item | cut -d '=' -f1)
-       value=$(echo $item | cut -d '=' -f2)
-       [ -n "$key" ] && service["$key"]=$value
-   done
-}
-
-function get_current_config()
-{
-   juju get ${service[service_name]} > /tmp/${service[service_name]}.cfg
-   config_info=$(python ${CWD}/parse_config.py /tmp/${service[service_name]}.cfg)
-   config_list=( $config_info )
-   for item in "${config_list[@]}"
-   do
-       key=$(echo $item | cut -d '=' -f1)
-       value=$(echo $item | cut -d '=' -f2)
-       [ -n "$key" ] && config["$key"]=$value
-   done
-   current_signature="${config[supportnodes]}-${config[roxienodes]}-${config[thornodes]}-${config[slavesPerNode]}"
-}
-
-
-function display_current_configuration()
-{
-   printf "%s%-15s: %8d\n" "$INDENT" "Support nodes" ${config[supportnodes]}
-   printf "%s%-15s: %8d\n" "$INDENT" "Roxie nodes" ${config[roxienodes]}
-   printf "%s%-15s: %8d\n" "$INDENT" "Thor nodes" ${config[thornodes]}
-   printf "%s%-15s: %8d\n" "$INDENT" "Slaves per node" ${config[slavesPerNode]}
-
-   #envgen_status="Current environment.xml probably already has above settings"
-   #if [ "$current_signation" != "${config[slavesPerNode]}" ]
-   #then
-   #   envgen_status="Current environment.xml probably doesnot have above settings"
-   #fi
-   #printf "%s%-15s: %s\n"  "$INDENT" "Status" "$envgen_status"
-
-   printf "\n"
-
-}
-
-function update_value()
-{
-   original_value=$1
-   [ -z "$2" ] && echo "$original_value" && return
-
-   input_value=$2
-
-   echo $input_value | grep -q -e "^[[:digit:]]"
-   [ $? -eq 0 ] && echo "${input_value}" && return
-
-   echo $input_value | grep -q -e "^+"
-   [ $? -eq 0 ] &&  input_value=${input_value:1}
-
-   echo "$(expr $original_value \+ ${input_value})"
-
-}
-
-function update_configuration()
-{
-   printf "%s%-15s  %8s %8s\n" "$INDENT" "ENVGEN OPTIONS" "BEFORE" "NOW"
-   printf "%s%s\n" "$INDENT" "----------------------------------"
-
-   supportnodes=$(update_value ${config[supportnodes]}  ${inputs[supportnodes]})
-   printf "%s%-15s: %8d %8d\n" "$INDENT" "Support nodes" \
-           ${config[supportnodes]} $supportnodes
-   config[supportnodes]=$supportnodes
-
-   roxienodes=$(update_value ${config[roxienodes]}  ${inputs[roxienodes]})
-   printf "%s%-15s: %8d %8d\n" "$INDENT" "Roxie nodes" \
-           ${config[roxienodes]} $roxienodes
-   config[roxienodes]=$roxienodes
-
-   thornodes=$(update_value ${config[thornodes]}  ${inputs[thornodes]})
-   printf "%s%-15s: %8d %8d\n" "$INDENT" "Thor nodes" \
-           ${config[thornodes]} $thornodes
-   config[thornodes]=$thornodes
-
-   slavesPerNode=$(update_value ${config[slavesPerNode]}  ${inputs[slavesPerNode]})
-   printf "%s%-15s: %8d %8d\n" "$INDENT" "Slaves Per Node" \
-           ${config[slavesPerNode]} $slavesPerNode
-   config[slavesPerNode]=$slavesPerNode
-
-   printf "\n"
-
-   if [ $update_only -eq 0 ]
-   then
-      config[envgen_signature]="${config[supportnodes]}-${config[roxienodes]}-${config[thornodes]}-${config[slavesPerNode]}"
-   fi
-
-}
-
-##
-## Main
-##################
-
-CWD=$(dirname $0)
-
-CUR_DIR=$(pwd)
-cd $CWD
-ABS_CWD=$(pwd)
-cd $CUR_DIR
-
-CHARM_NAME=$(basename $(dirname $ABS_CWD))
-
-
-declare -A inputs
-inputs[thornodes]=
-inputs[roxienodes]=
-inputs[supportnodes]=
-inputs[slavesPerNode]=
-
-list_only=0
-update_only=0
-
-declare -A config
-declare -A service
-service[service_name]=
-
-INDENT="    "
-
-
-num_args=$#
-for ((i=1; i<=num_args; i++))
-do
-  case $1 in
-    -name) shift
-            i=$(expr $i \+ 1)
-            service[service_name]=$1
-            ;;
-    -thornodes) shift
-            i=$(expr $i \+ 1)
-            inputs[thornodes]=$1
-            ;;
-    -roxienodes) shift
-            i=$(expr $i \+ 1)
-            inputs[roxienodes]=$1
-            ;;
-    -supportnodes) shift
-            i=$(expr $i \+ 1)
-            inputs[supportnodes]=$1
-            ;;
-    -slavespernode) shift
-            i=$(expr $i \+ 1)
-            inputs[slavesPerNode]=$1
-            ;;
-    -list) list_only=1
-            ;;
-    -updateonly) update_only=1
-            ;;
-    *) usage
-       exit 0
-  esac
-  shift
-done
-
-get_service_information
-get_current_config
-
-#for key in "${!inputs[@]}"
-#do
-#   echo "$key : ${inputs[$key]}"
-#done
-
-
-#echo "list_only=$list_only"
-#echo "update_only=$update_only"
-
-printf "\n"
-printf "%s%-15s: %-20s\n" "$INDENT" "CHARM NAME" "$CHARM_NAME"
-printf "%s%-15s: %-20s\n" "$INDENT" "SERVICE NAME" "${service[service_name]}"
-printf "%s%-15s: %-20s\n" "$INDENT" "UNIT NUMBER" "${service[unit_number]}"
-printf "\n"
-
-if [ $list_only -eq 1 ]
-then
-   display_current_configuration
-   exit
-fi
-
-
-update_configuration
-
-
-juju set ${service[service_name]} \
-           envgen-signature=${config[envgen_signature]} \
-           thornodes=${config[thornodes]} \
-           roxienodes=${config[roxienodes]} \
-           supportnodes=${config[supportnodes]} \
-           slavesPerNode=${config[slavesPerNode]}

charm/precise/hpcc/bin/get_url.sh

@@ -1,71 +0,0 @@
-#!/bin/bash
-
-
-function get_service_information()
-{
-
-
-   service_info=$(python ${ABS_CWD}/parse_status.py)
-   service_list=( $service_info )
-   for item in "${service_list[@]}"
-   do
-       key=$(echo $item | cut -d '=' -f1)
-       value=$(echo $item | cut -d '=' -f2)
-       if [ "$key" = "service_name" ]
-       then
-         service_name=$value
-         break
-       fi
-   done
-}
-
-
-##
-## Main
-##################
-
-CWD=$(dirname $0)
-
-CUR_DIR=$(pwd)
-cd $CWD
-ABS_CWD=$(pwd)
-cd $CUR_DIR
-
-CHARM_NAME=$(basename $(dirname $ABS_CWD))
-
-
-unit_name=$1
-service_name=
-
-eclwatch_url_file=eclwatch_url.txt
-LOCAL_ENV_FILE=/tmp/environment.xml
-LOCAL_URL_FILE=/tmp/${eclwatch_url_file}
-
-[ -e $LOCAL_ENV_FILE ] && rm -rf  $LOCAL_ENV_FILE
-[ -e $LOCAL_URL_FILE ] && rm -rf  $LOCAL_URL_FILE
-
-
-if [ -z "$unit_name" ]
-then
-  get_service_information
-  unit_name=$(juju status | grep "${service_name}/" | \
-    /usr/bin/head -n 1 | cut -d: -f1)
-fi
-
-juju scp ${unit_name}:/etc/HPCCSystems/environment.xml $LOCAL_ENV_FILE  > /dev/null 2>&1
-
-juju scp ${unit_name}:/var/lib/HPCCSystems/charm/${eclwatch_url_file} $LOCAL_URL_FILE > /dev/null 2>&1
-
-if [ -e $LOCAL_URL_FILE ]
-then
-   echo "ECLWatch URL: $(cat $LOCAL_URL_FILE)"
-else
-   echo "Failed to get $eclwatch_url_file from unit ${unit_name}"
-fi
-
-if [ -e $LOCAL_ENV_FILE ]
-then
-   echo "HPCC environment.xml is available under /tmp/"
-else
-   echo "Failed to get environment.xml from unit ${unit_name}"
-fi

charm/precise/hpcc/bin/parse_config.py

@@ -1,20 +0,0 @@
-import sys
-import yaml
-
-config_file = sys.argv[1]
-stream = file(config_file, "r")
-config = yaml.load(stream)
-
-envgen_options = [ 'roxienodes',
-                   'slavesPerNode',
-                   'supportnodes',
-                   'thornodes',
-                   'envgen-signature'
-                 ]
-
-for option in envgen_options:
-  try:
-      value = config["settings"][option]["value"]
-  except:
-      value = ""
-  print (option + "=" + str(value))

charm/precise/hpcc/bin/parse_status.py

@@ -1,121 +0,0 @@
-import sys
-import os.path
-import yaml
-import getopt
-import re
-#from subprocess import call
-
-
-def Usage():
-    print("\nUsage parse_status.py [options]\n")
-    print("  -?, --help      print help")
-    print("  -f, --file      Juju status file name. This is optional")
-    print("  -q, --query     Query information from juju status. Multiple")
-    print("                  query can be requested and seperated by comma")
-    print("  -s, --service   Charm service name. In chance deployed multiple")
-    print("                  service names for difference HPCC cluster this")
-    print("                  option should be provided, otherwise first service")
-    print("                  name will be used.")
-    print("\n")
-
-
-
-def LoadYamlObject():
-   global config
-   global status_file_name
-
-   stream = file(status_file_name, "r")
-   config = yaml.load(stream)
-
-def GetServiceName():
-   global service_name
-   global charm_name
-
-   if (service_name != "" ):
-      return
-   pattern = re.compile(".*\/"+charm_name+"-\d+")
-   for key in config["services"].keys():
-      if (pattern.match(config["services"][key]["charm"])):
-         service_name = key
-         print("service_name=" + key)
-         break
-
-def GetNumOfUnites():
-   global service_name
-
-   if (service_name == ""):
-      GetServiceName()
-
-   if (service_name == ""):
-      return
-
-   number_of_units = len(config["services"][service_name]["units"].keys())
-   print("unit_number=" + str(number_of_units))
-
-def ParseStatus():
-   global query_actions
-   global query_all
-
-   for action in query_all:
-       if (action == "service"):
-           GetServiceName()
-       elif (action == "num_of_units"):
-           GetNumOfUnites()
-
-
-
-
-if __name__ == "__main__":
-
-   charm_name= os.path.basename(
-               os.path.dirname(
-               os.path.dirname(
-               os.path.realpath(__file__))))
-
-   config =  None
-
-   query_all     = [
-                     "service",
-                     "num_of_units",
-                   ]
-   query_actions = ""
-   query_list = []
-   service_name = ""
-   status_file_name = ""
-
-   try:
-      opts, args = getopt.getopt(sys.argv[1:],":q:f:s:",
-           ["help", "query", "file", "service"])
-
-   except getopt.GetoptError as err:
-      print(str(err))
-      Usage()
-      exit()
-
-   for arg, value in opts:
-      if arg in ("-?", "--help"):
-         Usage()
-         exit(0)
-      elif arg in ("-q", "--query"):
-         query_actions = value
-      elif arg in ("-s", "--service"):
-         service_name = value
-      elif arg in ("-f", "--file"):
-         status_file_name = value
-      else:
-         print("\nUnknown option: " + arg)
-         Usage()
-         exit(0)
-
-   if ( query_actions != "" ):
-       query_list =  query_actions.split(',')
-   else:
-       query_list =  query_all
-
-   if ( status_file_name == "" ):
-      status_file_name = "/tmp/juju_status.txt"
-      os.system("juju status > " + status_file_name)
-
-   LoadYamlObject()
-
-   ParseStatus()

charm/precise/hpcc/config.yaml

@@ -1,111 +0,0 @@
-options:
-
-  base-url:
-    type: string
-    default: http://cdn.hpccsystems.com/releases
-    description: HPCC download base URL
-
-  hpcc-version:
-    type: string
-    default: 4.2.0-4
-    description: HPCC package version
-
-  hpcc-type:
-    type: string
-    default: CE
-    description: HPCC package type -  CE (Community Edition) is only choice now.
-
-  package-checksum:
-    type: string
-    default: bc2fb0ad5f9fa1eaf7a33483ec31bfa3
-    description: |
-      HPCC package checksum. The default the value is for HPCC CE 4.2.0-4 Ubunut
-      12.04 amd64. User can get other package checksum from:
-           http://hpccsystems.com/download
-      In future HPCCSystem download site will provide checksum file which will
-      allow automatically verification without this setting.
-      Set this value to empty string to skip the validation when download the
-      package.
-   
-  start-check-delay:
-    type: int
-    default: 5
-    description: |
-      Delay time in seconds. If the value is not zero it will wait amount of time
-      before checking the HPCC processes status. Otherwise will check immediately
-      after start.
-
-  download-url:
-    type: string
-    default: ""
-    description: |
-      Download URL. If this is defined all above three download related settings 
-      are ignored.
-
-  ssh-key-public:
-    type: string
-    default: ""
-    description: The content of public SSH Key
-
-  ssh-key-private:
-    type: string
-    default: ""
-    description: The content of private SSH Key
-
-
-
-# HPCC environment variable - Do we want to define them here?
-
-# Control when to run envgen
-  auto-gen:
-    type: int
-    default: 1
-    description: |
-      0: will not generate HPCC cluster configuration in relation-changed hook.
-         User need manually run bin/config_hpcc.sh script to do it.
-      1: automatically generate HPCC cluster configuration in relation-changed
-         hook with following topology with nodes N:
-           1 support nodes, N-1 thor nodes, and N-1 roxie nodes.
-      2: automatically generate HPCC cluster configuration in relation-changed
-         hook with parameters defined in service configuration. Usually user
-         run bin/config_hpcc.sh script with option -updateonly to set parameters
-         before add nodes to the cluster.
-
-
-# Following variables should not be changed directly by end users
-
-  envgen-signature:
-    type: string
-    default: ""
-    description: |
-      When manually trigger envgen set this variable as:
-        <supportnodes>-<roxienodes>-<thornodes>-<slavePerNode>
-      This will prevent unwanted running of envgen when roxienode/thornodes increase with new node join.
-      Each node also will keep previous envgen-sigature value in a file
-      Only previous value is different than current envgen-signature
-      will trigger envgen  from config-changed. It is for non-envgen related config variable update,(for example: download-url for HPCC update).
-      In this case, we don't want to trigger envgen.
-
-
-# Following are the parameters for envgen which will create environment.xml
-# Reference HPCC Platform envgen usage for more details
-
-  supportnodes:
-    type: int
-    default: 1
-    description: Number of nodes to be used for non-Thor and non-Roxie components.
-
-  roxienodes:
-    type: int
-    default: 1
-    description: Number of nodes to be generated for roxie.
-
-  thornodes:
-    type: int
-    default: 1
-    description: Number of nodes to be generated for thor.
-
-  slavesPerNode:
-    type: int
-    default: 0
-    description: Number of thor slaves per node.

charm/precise/hpcc/copyright

@@ -1,16 +0,0 @@
-[ Please note: HPCC SYSTEMS now licenses the HPCC Systems Community Edition, starting with v3.8, under the Apache v2 license agreement.  Appropriate files are being updated. ]
-
-
-HPCC SYSTEMS software Copyright (C) 2012 HPCC Systems.
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

charm/precise/hpcc/dependences/el6

@@ -1 +0,0 @@
-yum install -y boost-regex openldap libicu m4 libtool libxslt libxml2 gcc-c++ openssh-server openssh-clients expect libarchive rsync

charm/precise/hpcc/dependences/precise

@@ -1,3 +0,0 @@
-#!/bin/bash
-
-DEBIAN_FRONTEND=noninteractive apt-get -y install libboost-regex1.46.1 libicu48 libxalan110 libxerces-c28 binutils libldap-2.4-2 openssl zlib1g g++ openssh-client openssh-server expect libarchive12 rsync

charm/precise/hpcc/dependences/saucy

@@ -1,3 +0,0 @@
-#!/bin/bash
-
-DEBIAN_FRONTEND=noninteractive apt-get -y install libboost-regex1.53.0 libicu48 libxalan-c111 libxerces-c3.1 binutils libldap-2.4-2 openssl zlib1g g++ openssh-client openssh-server expect libarchive13 rsync

charm/precise/hpcc/hooks/config-changed

@@ -1,31 +0,0 @@
-#!/bin/bash -e
-# config-changed occurs everytime a new configuration value is updated (juju set)
-
-. ${CHARM_DIR}/hooks/hpcc-common
-
-##
-## Check if need to generate environment.xml
-##
-
-envgen_signature=$(config-get envgen-signature)
-juju-log "envgen_signature: $envgen_signature"
-[ -z "${envgen_signature}" ] && exit 0
-
-SIGNATURE_FILE=${JUJU_HPCC_DIR}/env_signature.txt
-if [ -e ${SIGNATURE_FILE} ]
-then
-    saved_signature=$(head -n 1 $SIGNATURE_FILE)
-    juju-log "saved_signature: $saved_signature"
-    [ "$saved_signature" == "$envgen_signature" ] && exit 0
-fi
-
-[ ! -e $IP_FILE ] && exit 0
-
-get_envgen_configuration
-
-composed_signature="${support_nodes}-${roxie_nodes}-${thor_nodes}-${slaves_per_node}"
-juju-log "composed_signature: $composed_signature"
-[ "$composed_signature" != "$envgen_signature" ] && exit 0
-
-
-configure_cluster

charm/precise/hpcc/hooks/hpcc-cluster-relation-changed

@@ -1,102 +0,0 @@
-#!/bin/bash -e
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by relationships being formed, modified, or broken
-# This script should be idempotent.
-
-
-. ${CHARM_DIR}/hooks/hpcc-common
-
-#juju-log $JUJU_REMOTE_UNIT modified its settings
-#juju-log Relation settings:
-#relation-get
-#juju-log Relation members:
-#relation-list
-
-TMP_FILE=${JUJU_HPCC_DIR}/ip_file.tmp
-
-LOCAL_IP=$(head -n 1  $LOCAL_IP_FILE) 
-echo "${LOCAL_IP};,$JUJU_UNIT_NAME" > ${TMP_FILE}
-
-RESET_SSH_KEYS=$(relation-get reset-ssh-keys)
-
-juju-log "RESET_SSH_KEYS: $RESET_SSH_KEYS"
-
-TOTAL_UNITS=1
-for UNIT in $(relation-list)
-do
-   UNIT_ADDRESS=$(relation-get netAddress ${UNIT})
-   if [ -n "${UNIT_ADDRESS}" ]
-   then
-      TOTAL_UNITS=$(expr ${TOTAL_UNITS} \+ 1)
-      echo "${UNIT_ADDRESS};,${UNIT}" >> "${TMP_FILE}"
-   fi
-done
-
-##
-## Sort IPs
-##
-cat $TMP_FILE | cut -d',' -f 1 | \
-   sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4  > $IP_FILE
-
-LOWEST_IP=$(head -n 1 $IP_FILE | cut -d ';' -f1)
-
-
-##
-## Set SSH Keys
-##
-if [ -n "$RESET_SSH_KEYS" ] &&  [ $RESET_SSH_KEYS -eq 1 ] 
-then
-  if [ "$LOWEST_IP" = "$LOCAL_IP" ]
-  then
-     PUB_KEY=$(relation-get ssh-key-public)
-     PRI_KEY=$(relation-get ssh-key-private)
-     if [ -z "$PUB_KEY" ] || [ -z "$PRI_KEY" ] 
-     then
-        create_and_set_ssh_keys
-     fi
-  else
-     LOWEST_IP_UNIT=$(cat ${TMP_FILE} | grep "${LOWEST_IP}" | \
-        cut -d',' -f 2)
-     SELECTED_PUB_KEY=$(relation-get ssh-key-public ${LOWEST_IP_UNIT})
-     SELECTED_PRI_KEY=$(relation-get ssh-key-private ${LOWEST_IP_UNIT})
-     [ -z "$SELECTED_PUB_KEY" ] || [ -z "$SELECTED_PRI_KEY" ] && exit 0 
-
-     # If we have the same public key as DESIRED_PUB_KEY
-     # our ssh keys are already set correctly
-     PUB_KEY=$(relation-get ssh-key-public)
-     if [ "$PUB_KEY" != "$SELECTED_PUB_KEY" ]
-     then 
-        juju-log "Set ssh keys from lowest ip system"
-        PUB_KEY=$SELECTED_PUB_KEY
-        PRI_KEY=$SELECTED_PRI_KEY
-        set_ssh_keys
-        relation-set ssh-key-public="${PUB_KEY}"
-        relation-set ssh-key-private="${PRI_KEY}"
-     fi
-  fi
-fi
-
-
-
-[  ${TOTAL_UNITS}  -eq 1 ] && exit 0
-
-##
-## If auto-gen set to 0 we need manually trigger
-#  configure_cluster by running bin/config_hpcc.sh script later
-##
-auto_gen=$(config-get auto-gen)
-juju-log "auto_gen $auto_gen"
-[  ${auto_gen}  -eq 0 ] && exit 0
-
-
-if [ ${auto_gen} -eq 1 ]
-then
-   configure_cluster ${TOTAL_UNITS}
-   rc=$?
-else
-   get_envgen_configuration
-   configure_cluster
-   rc=$?
-fi
-
-exit $rc

charm/precise/hpcc/hooks/hpcc-cluster-relation-departed

@@ -1,56 +0,0 @@
-#!/bin/bash -e
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by the remote unit leaving the relationship.
-# This script should be idempotent.
-juju-log $JUJU_REMOTE_UNIT departed
-
-
-JUJU_HPCC_DIR=/var/lib/HPCCSystems/charm
-TMP_FILE=${JUJU_HPCC_DIR}/ip_file.tmp
-
-LOCAL_IP_FILE=${JUJU_HPCC_DIR}/local_ip.txt
-IP_FILE=${JUJU_HPCC_DIR}/ip_file.txt
-
-[ -e ${LOCAL_IP_FILE} ] && echo "$(head -n 1  $LOCAL_IP_FILE);" > ${TMP_FILE}
-
-TOTAL_UNITS=1
-for UNIT in $(relation-list)
-do
-   UNIT_ADDRESS=`relation-get netAddress ${UNIT}`
-   if [ -n "${UNIT_ADDRESS}" ]
-   then
-      TOTAL_UNITS=$(expr ${TOTAL_UNITS} \+ 1)
-      echo "${UNIT_ADDRESS};" >> "${TMP_FILE}"
-   fi
-done
-
-[  ${TOTAL_UNITS}  -eq 1 ] && exit 0
-
-
-
-
-##
-## Sort IPs
-##
-cat $TMP_FILE | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4  > $IP_FILE
-
-##
-## DON'T HAVE A WAY TO CHECK IF CURRENT NODE IS LEAVING
-## IF IT IS FOLLOWING WILL FAIL.
-## For user need manually run  bin/config_hpcc.sh to reconfigure HPCC
-## after destroy a unit.
-##
-
-#. ${CHARM_DIR}/hooks/hpcc-common
-
-#auto_gen=$(config-get auto-gen)
-#juju-log "auto_gen $auto_gen"
-#[  ${auto_gen}  -eq 0 ] && exit
-
-#if [ ${auto_gen} -eq 1 ]
-#then
-#   configure_cluster ${TOTAL_UNITS}
-#else
-#   get_envgen_configuration
-#   configure_cluster
-#fi

charm/precise/hpcc/hooks/hpcc-cluster-relation-joined

@@ -1,22 +0,0 @@
-#!/bin/bash -e
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by relationships being formed
-# This script should be idempotent.
-juju-log $JUJU_REMOTE_UNIT joined
-
-. ${CHARM_DIR}/hooks/hpcc-common
-
-save_local_ip
-
-relation-set name="${HOSTNAME}" netAddress="${IP_ADDRESS}"
-
-PUB_KEY=$(config-get ssh-key-public)
-PRI_KEY=$(config-get ssh-key-private)
-fix_private_key
-if [ -z "$PUB_KEY" ] || [ -z "$PUB_KEY" ]
-then
-   relation-set reset-ssh-keys=1
-   create_and_set_ssh_keys
-else
-   relation-set reset-ssh-keys=0
-fi

charm/precise/hpcc/hooks/hpcc-common

@@ -1,229 +0,0 @@
-JUJU_HPCC_DIR=/var/lib/HPCCSystems/charm
-CONFIG_DIR=/etc/HPCCSystems
-ENV_XML_FILE=environment.xml
-ENV_CONF_FILE=environment.conf
-ENV_GEN="/opt/HPCCSystems/sbin/envgen"
-SIGNATURE_FILE=${JUJU_HPCC_DIR}/env_signature.txt
-LOCAL_IP_FILE=${JUJU_HPCC_DIR}/local_ip.txt
-IP_FILE=${JUJU_HPCC_DIR}/ip_file.txt
-ECLWATCH_URL_FILE=${JUJU_HPCC_DIR}/eclwatch_url.txt
-
-##
-## Fix private key in chance newlines removed
-##
-function fix_private_key()
-{
-   head_end_marks=$(echo "${PRI_KEY}" | \
-       sed -n "s/\(-----.*-----\).*\(-----.*-----\)/\1,\2/p" )
-
-   [ -z "$head_end_marks" ] && return
-
-   BEGIN_KEY=$(echo ${head_end_marks} | cut -d',' -f1)
-   END_KEY=$(echo ${head_end_marks} | cut -d',' -f2)
-
-   private_key=$(echo "${PRI_KEY}" | \
-       sed "s/${BEGIN_KEY}\(.*\)${END_KEY}/\1/g" | sed 's/ /\n/g')
-
-   END_KEY=$(echo "${END_KEY}"  | sed 's/^/\n/g')
-   PRI_KEY="${BEGIN_KEY}${private_key}${END_KEY}"
-
-}
-
-##
-## Parse environment.conf
-##
-function parse_env_config()
-{
-  HPCC_CONFIG=${HPCC_CONFIG:-${CONFIG_DIR}/${ENV_CONF_FILE}}
-  SECTION=${1:-DEFAULT}
-
-  PATH_PREFIX=$(cat ${HPCC_CONFIG} | sed -n "/\[${SECTION}\]/,/\[/p" | \
-     grep "^home *= *" | sed -e 's/^home *= *//')
-  USER_NAME=$(cat ${HPCC_CONFIG} | sed -n "/\[${SECTION}\]/,/\[/p" | \
-     grep "^user *= *" | sed -e 's/^user *= *//')
-
-  PATH_HOME=${PATH_PREFIX}/${USER_NAME}
-}
-
-##
-## Set SSH Keys
-##
-function set_ssh_keys()
-{
-  [ -z "$PUB_KEY" ] || [ -z "$PRI_KEY" ] && return 0
-
-  parse_env_config
-  juju-log "Set SSH Keys. HOME: $PATH_HOME, USER: $USER_NAME."
-  if [ ! -d $PATH_HOME/.ssh ]; then
-     mkdir $PATH_HOME/.ssh
-  fi
-  rm -fr $PATH_HOME/.ssh/*
-
-
-  echo "$PUB_KEY" > $PATH_HOME/.ssh/id_rsa.pub
-  echo "$PRI_KEY" > $PATH_HOME/.ssh/id_rsa
-  if [ -e $PATH_HOME/.ssh/id_rsa.pub ]; then
-     cat $PATH_HOME/.ssh/id_rsa.pub > $PATH_HOME/.ssh/authorized_keys
-  fi
-
-  chown -R $USER_NAME:$USER_NAME $PATH_HOME/.ssh
-  chmod 644 $PATH_HOME/.ssh/authorized_keys
-  chmod 600 $PATH_HOME/.ssh/id_rsa
-  chmod 644 $PATH_HOME/.ssh/id_rsa.pub
-
-}
-
-##
-## Query/Create and set SSH Keys
-##
-function create_and_set_ssh_keys()
-{
-  for UNIT in $(relation-list)
-  do
-    PUB_KEY=$(relation-get ssh-key-public ${UNIT})
-    PRI_KEY=$(relation-get ssh-key-private ${UNIT})
-    [ -n "$PUB_KEY" ] && [ -n "$PRI_KEY" ]  && break
-  done
-  if [ -z "$PUB_KEY" ] || [ -z "$PRI_KEY" ]
-  then
-    SSH_DIR=/tmp/hpcc_ssh
-    [ -d ${SSH_DIR} ] && rm -rf ${SSH_DIR}
-    mkdir ${SSH_DIR}
-    juju-log "Generate ssh keys with ssh-keygen"
-    ssh-keygen -f ${SSH_DIR}/id_isa -t rsa -N ''
-    PUB_KEY=$(cat ${SSH_DIR}/id_isa.pub)
-    PRI_KEY=$(cat ${SSH_DIR}/id_isa)
-    rm -rf ${SSH_DIR}
-  fi
-  set_ssh_keys
-  if [ -n "${JUJU_RELATION_ID}" ]
-  then
-     relation-set ssh-key-public="${PUB_KEY}"
-     relation-set ssh-key-private="${PRI_KEY}"
-  fi
-}
-
-##
-## Save local ip to a file
-##
-function save_local_ip()
-{
-  HOSTNAME=$(unit-get public-address)
-  PRIVATE_HOSTNAME=$(unit-get private-address)
-  IP_ADDRESS=$(dig +short ${PRIVATE_HOSTNAME} | tail -n 1)
-  echo "$IP_ADDRESS" > $LOCAL_IP_FILE
-}
-
-##
-## Get currently defined envgen parameters
-##
-function get_envgen_configuration()
-{
-  support_nodes=$(config-get supportnodes)
-  thor_nodes=$(config-get thornodes)
-  roxie_nodes=$(config-get roxienodes)
-  slaves_per_node=$(config-get slavesPerNode)
-}
-
-
-##
-## Generate environemt.xml
-##
-function configure_cluster()
-{
-
-  number_of_nodes=$1
-
-  if [ -n "$number_of_nodes" ]
-  then
-     support_nodes=1
-     thor_nodes=$(expr $number_of_nodes \- 1)
-     roxie_nodes=$(expr $number_of_nodes \- 1)
-     slaves_per_node=1
-  fi
-
-  juju-log "Stop HPCC"
-  # Following command return 3 even stop succeeds
-  # It will be fixed in HPCC 5.0 release
-  service hpcc-init stop || :
-
-  juju-log "Run envgen ${ENV_GEN} -env ${CONFIG_DIR}/${ENV_XML_FILE} -ipfile ${IP_FILE} -thornodes ${thor_nodes} -slavesPerNode ${slaves_per_node} -roxienodes ${roxie_nodes} -supportnodes ${support_nodes}"
-  "${ENV_GEN}" -env "${CONFIG_DIR}/${ENV_XML_FILE}" -ipfile "${IP_FILE}" -thornodes "${thor_nodes}" -slavesPerNode "${slaves_per_node}" -roxienodes "${roxie_nodes}" -supportnodes "${support_nodes}"
-  [ $? -ne 0 ] && exit 1
-
-  juju-log "Start HPCC"
-  start_and_check
-  rc=$?
-
-  save_evngen_signature
-  save_eclwatch_url
-
-  return $rc
-}
-
-##
-## Save envgen signature
-##
-function save_evngen_signature()
-{
-
-   _signature="${support_nodes}-${roxie_nodes}-${thor_nodes}-${slaves_per_node}"
-   juju-log "Save envgen signature: $_signature"
-   echo "$_signature" > $SIGNATURE_FILE
-}
-
-
-##
-## Save EclWatch URL to a file
-##
-function save_eclwatch_url()
-{
-  juju-log "Update Check ECLWatch URL"
-
-  first_node=$(head -n 1 $IP_FILE | cut -d';' -f1)
-  cat ${CONFIG_DIR}/${ENV_XML_FILE} | grep -q $first_node || exit 0
-
-  eclwatch_port=$(cat ${CONFIG_DIR}/${ENV_XML_FILE} | \
-  sed -n '
-    /<EspBinding/,/>/ {
-      s/port=\([[:digit:]]*\)/\1/p
-    }
-  ' | head -n 1 | cut -d '"' -f 2)
-
-  eclwatch_ip=$(cat ${CONFIG_DIR}/${ENV_XML_FILE} | \
-  sed -n '
-    /<EspProcess/,/EspProcess>/ {
-      s/netAddress=\([[:digit:]]*\)/\1/p
-    }
-  ' |  cut -d '"' -f 2)
-
-  juju-log "set eclwatch-url http://${eclwatch_ip}:${eclwatch_port}"
-
-  echo "http://${eclwatch_ip}:${eclwatch_port}" > ${ECLWATCH_URL_FILE}
-}
-
-##
-## Check HPCC Status
-##
-function check_hpcc_status()
-{
-  stopped_components=$(service hpcc-init status | grep "stopped" | awk '{ print $1 }')
-  [ -z "$stopped_components" ] && return 0
-  juju-log "Failed to start HPCC processes. Check each log directory under /var/log/HPCCSysgtems for detail: ${stopped_components}."
-  return 1
-}
-
-##
-## Start HPCC and check its status
-##
-function start_and_check()
-{
-
-  juju-log "Starting HPCC ..."
-  service hpcc-init start
-  start_check_delay=$(config-get start-check-delay)
-  [ ${start_check_delay} -gt 0 ] && sleep ${start_check_delay}
-
-  check_hpcc_status || return $?
-  
-}

charm/precise/hpcc/hooks/install

@@ -1,165 +0,0 @@
-#!/bin/bash -e
-
-##
-## Set debugging
-##
-set -ux
-
-
-##
-## Current working directory
-##
-CWD=$(dirname $0)
-
-. ${CHARM_DIR}/hooks/hpcc-common
-
-
-##
-## Detect Linux Distro
-##
-uname -a  | grep -q "x86_64" || ( juju-log "Only arch x86_64 is supported."; exit 1)
-
-
-Distributor_ID=$(lsb_release -a | grep  "Distributor ID:" | \
-sed -n "s/.*:[[:space:]]*\(.*\)/\1/p")
-juju-log "Distriobutor_ID: $Distributor_ID"
-
-if [ "$Distributor_ID" == "CentOS" ]
-then
-   os_release=$(lsb_release -a | grep -e "Release:" | \
-   sed -n "s/.*:[[:space:]]*\(.*\)/\1/p")
-   if [ ${os_release:0:1} -eq 6 ]
-   then
-      os_id=el6
-   elif [ ${os_release:0:1} -eq 5 ]
-   then
-      os_id=el5
-   else
-      juju-log "Unsupported CenOS version: ${os_release}."
-      exit 1
-   fi
-
-elif [ "$Distributor_ID" == "Ubuntu" ]
-then
-   code_name=$(lsb_release -a | grep -e "Codename:" | \
-   sed -n "s/.*:[[:space:]]*\(.*\)/\1/p")
-   os_id=$code_name
-else
-   juju-log "Unsupported Linux for Juju/charm HPCC: $Distributor_ID"
-   exit 1
-fi
-juju-log "OS release version:  $os_id"
-
-
-##
-## HPCC prerequisites
-##
-dependence_dir=${CWD}/../dependences
-if [ ! -e ${dependence_dir}/${os_id} ]
-then
-   juju-log "Prequisites for $os_id hasn't been implemented yet."
-   exit 1
-fi
-${dependence_dir}/${os_id}  || exit 1
-
-
-##
-## Download and install HPCC
-##
-download_url=$(config-get download-url)
-if [ -z "$download_url" ]
-then
-  base_url=$(config-get  base-url)
-  hpcc_type=$(config-get  hpcc-type)
-  hpcc_version=$(config-get  hpcc-version)
-  hpcc_version_mmp=$(echo $hpcc_version | cut -d '-' -f1)
-  juju-log "hpcc_version_mmp: $hpcc_version_mmp"
-
-  declare -A type_detail
-  type_detail[CE]=community
-  type_detail[EE]=enterprise
-  type_detail[LN]=internal
-
-  package_name="hpccsystems-platform_${type_detail[$hpcc_type]}-${hpcc_version}"
-  download_url="${base_url}/${hpcc_type}-Candidate-${hpcc_version_mmp}"
-  download_url="${download_url}/bin/platform/$package_name"
-
-  if [ "$Distributor_ID" == "CentOS" ]
-  then
-    download_url="${download_url}.${os_id}_x86_64.rpm"
-  else
-     download_url="${download_url}${os_id}_amd64.deb"
-  fi
-fi
-
-juju-log "Download URL: $download_url"
-
-
-curl -s -O $download_url
-hpcc_package=$(basename $download_url)
-
-expected_checksum=$(config-get package-checksum)
-if [ -n "$expected_checksum" ]
-then 
-   downloaded_checksum=$(md5sum ${hpcc_package} | cut -d ' ' -f1)
-   if [ "$expected_checksum" != "${downloaded_checksum}" ]
-   then
-      juju-log "Package checksum don't match. Expected ${expected_checksum} but get ${downloaded_checksum}" 
-      exit 1
-   fi
-else
-  juju-log "Missing package-checksum in config.yaml. You must validate checksum before installing the HPCC package. The checksum can be retrieved from http://hpccsystems.com/download/free-community-edition/server-platform for HPCC Community Server Platform." 
-  exit 1
-   
-fi
-
-if [ "$Distributor_ID" == "CentOS" ]
-then
-   rpm -i $hpcc_package || exit 1
-else
-   DEBIAN_FRONTEND=noninteractive dpkg -i $hpcc_package || exit 1
-fi
-
-
-##
-## Set SSH keys
-## Using shipped for now. Will provide script to generate new kyes later
-##
-parse_env_config
-echo $PATH_HOME
-
-if [ ! -d $PATH_HOME/.ssh ]; then
-        mkdir $PATH_HOME/.ssh
-fi
-rm -fr $PATH_HOME/.ssh/*
-
-set +x
-PUB_KEY=$(config-get ssh-key-public)
-PRI_KEY=$(config-get ssh-key-private)
-fix_private_key
-set_ssh_keys
-set -x
-
-##
-## Open the necessory firewall ports
-##
-which open-port  && \
-if [ $? -eq 0 ];then
-   open-port 8010/TCP
-   open-port 8002/TCP
-   open-port 8015/TCP
-   open-port 9876/TCP
-   #open-port 18010/TCP
-   #open-port 18002/TCP
-   #open-port 18008/TCP
-fi
-
-##
-## Create directory for stagging and tempory files
-##
-JUJU_HPCC_DIR=/var/lib/HPCCSystems/charm
-if [ ! -e ${JUJU_HPCC_DIR} ]
-then
-   mkdir -p $JUJU_HPCC_DIR
-   chmod -R 777 $JUJU_HPCC_DIR
-fi

charm/precise/hpcc/hooks/start

@@ -1,18 +0,0 @@
-#!/bin/bash -e
-# Here put anything that is needed to start the service.
-# Note that currently this is run directly after install
-# i.e. 'service apache2 start'
-
-#export PS4='+${BASH_SOURCE[1]} ${LINENO}'
-#set -x
-
-. ${CHARM_DIR}/hooks/hpcc-common
-
-parse_env_config
-if [ ! -e ${PATH_HOME}/.ssh/id_rsa ]
-then
-   create_and_set_ssh_keys
-fi
-
-start_and_check
-exit $?

charm/precise/hpcc/hooks/stop

@@ -1,10 +0,0 @@
-#!/bin/bash -e
-# This will be run when the service is being torn down, allowing you to disable
-# it in various ways..
-# For example, if your web app uses a text file to signal to the load balancer
-# that it is live... you could remove it and sleep for a bit to allow the load
-# balancer to stop sending traffic.
-# rm /srv/webroot/server-live.txt && sleep 30
-
-# work-around current HPCC stop return code
-service hpcc-init stop || :

charm/precise/hpcc/metadata.yaml

@@ -1,33 +0,0 @@
-name: hpcc
-summary: HPCC (High Performance Computing Cluster)
-maintainer: xiaoming.wang@lexisnexis.com
-description: |
-  HPCC Systems, an open source High Performance Computing Cluster, is a massive parallel-processing computing 
-  platform that solves Big Data problems. HPCC Systems is an enterprise-proven platform for manipulating, 
-  transforming, querying, and data warehousing Big Data. Built by LexisNexis, the HPCC platform has helped 
-  it grow to a $1.5 billion information solutions company.
-  .
-  The HPCC Systems architecture incorporates a data query engine (called Thor) and a data delivery engine 
-  (called Roxie), as well as support components. An HPCC environment can include only Thor clusters, 
-  or both Thor and Roxie clusters. The HPCC Juju charm creates a cluster which contains both, but you can 
-  customize it after deployment.
-  .
-  The HPCC Juju Charm encapsulates best practice configurations for the HPCC Systems Platform. You can use a Juju Charm to stand up an HPCC Platform on:
-  .
-  * Local Provider (LXC)
-  * Amazon Web Services Cloud
-  
-  
-categories:
-    - applications
-subordinate: false
-
-provides:
-  hpcc:
-    interface: hpcc
-#requires:
-#  requires-relation:
-#    interface: interface-name
-peers:
-  hpcc-cluster:
-    interface: hpcc-cluster

charm/precise/hpcc/revision

@@ -1 +0,0 @@
-1

common/fileview2/fileview.hpp

@@ -202,9 +202,9 @@ extern FILEVIEW_API unsigned getResultBin(MemoryBuffer & ret, INewResultSet * cu
 #define WorkUnitXML_NoRoot          0x0002
 #define WorkUnitXML_SeverityTags    0x0004
 
-extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const char *password, const IConstWorkUnit *cw, IXmlWriter &writer, unsigned flags, WUExceptionSeverity minSeverity, const char *rootTag);
-extern FILEVIEW_API IStringVal& getFullWorkUnitResultsXML(const char *user, const char *pw, const IConstWorkUnit *wu, IStringVal &str, unsigned flags=0, WUExceptionSeverity minSeverity=ExceptionSeverityInformation);
-extern FILEVIEW_API IStringVal& getFullWorkUnitResultsJSON(const char *user, const char *pw, const IConstWorkUnit *wu, IStringVal &str, unsigned flags=0, WUExceptionSeverity minSeverity=ExceptionSeverityInformation);
+extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const char *password, const IConstWorkUnit *cw, IXmlWriter &writer, unsigned flags, ErrorSeverity minSeverity, const char *rootTag);
+extern FILEVIEW_API IStringVal& getFullWorkUnitResultsXML(const char *user, const char *pw, const IConstWorkUnit *wu, IStringVal &str, unsigned flags=0, ErrorSeverity minSeverity=SeverityInformation);
+extern FILEVIEW_API IStringVal& getFullWorkUnitResultsJSON(const char *user, const char *pw, const IConstWorkUnit *wu, IStringVal &str, unsigned flags=0, ErrorSeverity minSeverity=SeverityInformation);
 
 extern FILEVIEW_API void startRemoteDataSourceServer(const char * queue, const char * cluster);
 extern FILEVIEW_API void stopRemoteDataSourceServer();

common/fileview2/fverror.hpp

@@ -18,41 +18,44 @@
 #ifndef FVERROR_HPP
 #define FVERROR_HPP
 
-#include "jerrorrange.hpp"
+#include "errorlist.h"
 
-#define ERR_FILEVIEW_FIRST  2000
-#define ERR_FILEVIEW_LAST   2049
+//Range 6700..6749 is reserved
+#if (FILEVIEW_ERROR_START != 6700 || FILEVIEW_ERROR_END != 6749)
+#error "FILEVIEW_ERROR_START has changed"
+#endif
 
-#define FVERR_CouldNotResolveX                  2000
-#define FVERR_NoRecordDescription               2001
-#define FVERR_BadRecordDesc                     2002
-#define FVERR_NeedClusterToBrowseX              2003
-#define FVERR_TimeoutRemoteFileView             2004
-#define FVERR_UnknownRemoteCommand              2005
-#define FVERR_UnknownUTFFormat                  2006
-#define FVERR_FailedOpenFile                    2007
-#define FVERR_CompressedFile                    2008
-#define FVERR_CannotViewKey                     2009
-#define FVERR_ViewComplexKey                    2010
-#define FVERR_FilterTooRestrictive              2011
-#define FVERR_ZeroSizeRecord                    2012
-#define FVERR_FailedOpenCompressedFile          2013
-#define FVERR_UnrecognisedJoinFieldSyntax       2014
-#define FVERR_UnrecognisedJoinFieldSyntaxXX     2015
-#define FVERR_UnrecognisedMappingFunctionX      2016
-#define FVERR_UnrecognisedFieldX                2017
-#define FVERR_ExpectedFieldSelectedFromDatasetXX 2018
-#define FVERR_CannotSelectFromDatasetX          2019
-#define FVERR_CannotSelectManyFromDatasetX      2020
-#define FVERR_ExpectedFieldSelectedFromRecordXX 2021
-#define FVERR_NumJoinFieldsMismatchXY           2022
-#define FVERR_ExpectedX                         2023
-#define FVERR_FailTransformation                2024
-#define FVERR_UnrecognisedMappingFunctionXY     2025
-#define FVERR_BadStringTermination              2026
-#define FVERR_CannotBrowseFile                  2027
-#define FVERR_PluginMismatch                    2028
-#define FVERR_RowTooLarge                       2029
+#define FVERR_CouldNotResolveX                  6700
+#define FVERR_NoRecordDescription               6701
+#define FVERR_BadRecordDesc                     6702
+#define FVERR_NeedClusterToBrowseX              6703
+#define FVERR_TimeoutRemoteFileView             6704
+#define FVERR_UnknownRemoteCommand              6705
+#define FVERR_UnknownUTFFormat                  6706
+#define FVERR_FailedOpenFile                    6707
+#define FVERR_CompressedFile                    6708
+#define FVERR_CannotViewKey                     6709
+#define FVERR_ViewComplexKey                    6710
+#define FVERR_FilterTooRestrictive              6711
+#define FVERR_ZeroSizeRecord                    6712
+#define FVERR_FailedOpenCompressedFile          6713
+#define FVERR_UnrecognisedJoinFieldSyntax       6714
+#define FVERR_UnrecognisedJoinFieldSyntaxXX     6715
+#define FVERR_UnrecognisedMappingFunctionX      6716
+#define FVERR_UnrecognisedFieldX                6717
+#define FVERR_ExpectedFieldSelectedFromDatasetXX 6718
+#define FVERR_CannotSelectFromDatasetX          6719
+#define FVERR_CannotSelectManyFromDatasetX      6720
+#define FVERR_ExpectedFieldSelectedFromRecordXX 6721
+#define FVERR_NumJoinFieldsMismatchXY           6722
+#define FVERR_ExpectedX                         6723
+#define FVERR_FailTransformation                6724
+#define FVERR_UnrecognisedMappingFunctionXY     6725
+#define FVERR_BadStringTermination              6726
+#define FVERR_CannotBrowseFile                  6727
+#define FVERR_PluginMismatch                    6728
+#define FVERR_RowTooLarge                       6729
+#define FVERR_CouldNotProcessSchema             6730
 
 #define FVERR_CouldNotResolveX_Text             "Could not resolve file '%s' in DFS"
 #define FVERR_NoRecordDescription_Text          "DFS did not contain a record description for '%s'"

common/fileview2/fvresultset.cpp

@@ -3265,19 +3265,19 @@ extern FILEVIEW_API unsigned getResultBin(MemoryBuffer & ret, INewResultSet * re
     return getResultCursorBin(ret, cursor, start, count);
 }
 
-inline const char *getSeverityTagname(WUExceptionSeverity severity, unsigned flags)
+inline const char *getSeverityTagname(ErrorSeverity severity, unsigned flags)
 {
     if (flags & WorkUnitXML_SeverityTags)
     {
         switch (severity)
         {
-        case ExceptionSeverityInformation:
+        case SeverityInformation:
             return "Info";
-        case ExceptionSeverityWarning:
+        case SeverityWarning:
             return "Warning";
-        case ExceptionSeverityAlert:
+        case SeverityAlert:
             return "Alert";
-        case ExceptionSeverityError:
+        case SeverityError:
         default:
             break;
         }
@@ -3285,7 +3285,7 @@ inline const char *getSeverityTagname(WUExceptionSeverity severity, unsigned fla
     return "Exception";
 }
 
-extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const char *password, const IConstWorkUnit *cw, IXmlWriter &writer, unsigned flags, WUExceptionSeverity minSeverity, const char *rootTag)
+extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const char *password, const IConstWorkUnit *cw, IXmlWriter &writer, unsigned flags, ErrorSeverity minSeverity, const char *rootTag)
 {
     if (rootTag && *rootTag && !(flags & WorkUnitXML_NoRoot))
         writer.outputBeginNested(rootTag, true);
@@ -3294,7 +3294,7 @@ extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const ch
     ForEach(*exceptions)
     {
         IConstWUException & exception = exceptions->query();
-        WUExceptionSeverity severity = exception.getSeverity();
+        ErrorSeverity severity = exception.getSeverity();
         if (severity>=minSeverity)
         {
             SCMStringBuffer src, msg, filename;
@@ -3351,7 +3351,7 @@ extern FILEVIEW_API void writeFullWorkUnitResults(const char *username, const ch
         writer.outputEndNested(rootTag);
 }
 
-extern FILEVIEW_API IStringVal& getFullWorkUnitResultsXML(const char *username, const char *password, const IConstWorkUnit *cw, IStringVal &str, unsigned flags, WUExceptionSeverity minSeverity)
+extern FILEVIEW_API IStringVal& getFullWorkUnitResultsXML(const char *username, const char *password, const IConstWorkUnit *cw, IStringVal &str, unsigned flags, ErrorSeverity minSeverity)
 {
     Owned<CommonXmlWriter> writer = CreateCommonXmlWriter(XWFexpandempty);
     writeFullWorkUnitResults(username, password, cw, *writer, flags, minSeverity, "Result");
@@ -3364,7 +3364,7 @@ extern FILEVIEW_API IStringVal& getFullWorkUnitResultsXML(const char *username,
     return str;
 }
 
-extern FILEVIEW_API IStringVal& getFullWorkUnitResultsJSON(const char *username, const char *password, const IConstWorkUnit *cw, IStringVal &str, unsigned flags, WUExceptionSeverity minSeverity)
+extern FILEVIEW_API IStringVal& getFullWorkUnitResultsJSON(const char *username, const char *password, const IConstWorkUnit *cw, IStringVal &str, unsigned flags, ErrorSeverity minSeverity)
 {
     Owned<CommonJsonWriter> writer = new CommonJsonWriter(0);
     writer->outputBeginRoot();

common/fileview2/fvsource.cpp

@@ -1102,7 +1102,7 @@ bool FVDataSource::setReturnedInfoFromResult()
     wuResult->getResultEclSchema(s);
     returnedRecord.setown(parseQuery(s.str()));
     if (!returnedRecord)
-        throw MakeStringException(ERR_FILEVIEW_FIRST+4, "Could not process result schema [%s]", s.str());
+        throw MakeStringException(FVERR_CouldNotProcessSchema, "Could not process result schema [%s]", s.str());
 
     bool isKey = false;
     bool isGrouped = false;     // this isn't strictly true...it could be true for an internal result, but no current flag to test
@@ -1248,7 +1248,7 @@ IFvDataSourceMetaData * createMetaData(IConstWUResult * wuResult)
     wuResult->getResultEclSchema(s);
     OwnedHqlExpr record = parseQuery(s.str());
     if (!record)
-        throw MakeStringException(ERR_FILEVIEW_FIRST+4, "Could not process result schema [%s]", s.str());
+        throw MakeStringException(FVERR_CouldNotProcessSchema, "Could not process result schema [%s]", s.str());
 
     OwnedHqlExpr simplifiedRecord = getFileViewerRecord(record, false);
     bool isGrouped = false;     // more not sure this is strictly true...

common/thorhelper/roxiehelper.cpp

@@ -1464,9 +1464,9 @@ StringBuffer & mangleLocalTempFilename(StringBuffer & out, char const * in)
 
 static const char *skipLfnForeign(const char *lfn)
 {
+    while (*lfn=='~')
+        lfn++;
     const char *finger = lfn;
-    while (*finger=='~')
-        finger++;
     const char *scope = strstr(finger, "::");
     if (scope)
     {
@@ -1490,12 +1490,12 @@ static const char *skipLfnForeign(const char *lfn)
 
 StringBuffer & expandLogicalFilename(StringBuffer & logicalName, const char * fname, IConstWorkUnit * wu, bool resolveLocally)
 {
-    fname = skipLfnForeign(fname); //foreign location should already be reflected in local dali dfs meta data
+    const char *native = skipLfnForeign(fname); //foreign location should already be reflected in local dali dfs meta data
     if (fname[0]=='~')
-        logicalName.append(fname+1);
+        logicalName.append(native);
     else if (resolveLocally)
     {
-        StringBuffer sb(fname);
+        StringBuffer sb(native);
         sb.replaceString("::",PATHSEPSTR);
         makeAbsolutePath(sb.str(), logicalName.clear());
     }
@@ -1508,7 +1508,7 @@ StringBuffer & expandLogicalFilename(StringBuffer & logicalName, const char * fn
             if(lfn.length())
                 logicalName.append(lfn.s).append("::");
         }
-        logicalName.append(fname);
+        logicalName.append(native);
     }
     return logicalName;
 }

common/workunit/referencedfilelist.cpp

@@ -144,7 +144,7 @@ public:
 
     void reset()
     {
-        flags &= RefSubFile;
+        flags &= (RefSubFile | RefFileIndex | RefFileForeign | RefFileSuper | RefSubFile | RefFileInPackage);
     }
 
     IPropertyTree *getRemoteFileTree(IUserDescriptor *user, INode *remote, const char *remotePrefix);
@@ -295,7 +295,13 @@ void ReferencedFile::processRemoteFileTree(IPropertyTree *tree, const char *srcC
         {
             Owned<IPropertyTreeIterator> it = tree->getElements("SubFile");
             ForEach(*it)
-                subfiles->append(it->query().queryProp("@name"));
+            {
+                const char *lfn = it->query().queryProp("@name");
+                StringBuffer foreignLfn;
+                if (flags & RefFileForeign)
+                    lfn = foreignLfn.append("foreign::").append(this->daliip).append("::").append(lfn).str();
+                subfiles->append(lfn);
+            }
         }
     }
     else if (srcCluster && *srcCluster)

common/workunit/workunit.cpp

@@ -1813,7 +1813,7 @@ public:
     virtual IStringVal& getExceptionSource(IStringVal &str) const;
     virtual IStringVal& getExceptionMessage(IStringVal &str) const;
     virtual unsigned    getExceptionCode() const;
-    virtual WUExceptionSeverity getSeverity() const;
+    virtual ErrorSeverity getSeverity() const;
     virtual IStringVal & getTimeStamp(IStringVal & dt) const;
     virtual IStringVal & getExceptionFileName(IStringVal & str) const;
     virtual unsigned    getExceptionLineNo() const;
@@ -1821,7 +1821,7 @@ public:
     virtual void        setExceptionSource(const char *str);
     virtual void        setExceptionMessage(const char *str);
     virtual void        setExceptionCode(unsigned code);
-    virtual void        setSeverity(WUExceptionSeverity level);
+    virtual void        setSeverity(ErrorSeverity level);
     virtual void        setTimeStamp(const char * dt);
     virtual void        setExceptionFileName(const char *str);
     virtual void        setExceptionLineNo(unsigned r);
@@ -4759,10 +4759,11 @@ void CLocalWorkUnit::setSnapshot(const char * val)
 
 const static mapEnums warningSeverityMap[] =
 {
-    { SeverityIgnore, "ignore" },
-    { SeverityInfo, "info" },
+    { SeverityInformation, "info" },
     { SeverityWarning, "warning" },
     { SeverityError, "error" },
+    { SeverityAlert, "alert" },
+    { SeverityIgnore, "ignore" },
     { SeverityFatal, "fatal" },
     { SeverityUnknown, NULL }
 };
@@ -8168,9 +8169,9 @@ unsigned  CLocalWUException::getExceptionCode() const
     return p->getPropInt("@code", 0);
 }
 
-WUExceptionSeverity CLocalWUException::getSeverity() const
+ErrorSeverity CLocalWUException::getSeverity() const
 {
-    return (WUExceptionSeverity)p->getPropInt("@severity", ExceptionSeverityError);
+    return (ErrorSeverity)p->getPropInt("@severity", SeverityError);
 }
 
 IStringVal & CLocalWUException::getTimeStamp(IStringVal & dt) const
@@ -8210,7 +8211,7 @@ void CLocalWUException::setExceptionCode(unsigned code)
     p->setPropInt("@code", code);
 }
 
-void CLocalWUException::setSeverity(WUExceptionSeverity level)
+void CLocalWUException::setSeverity(ErrorSeverity level)
 {
     p->setPropInt("@severity", level);
 }
@@ -9559,7 +9560,7 @@ extern WORKUNIT_API IExtendedWUInterface * queryExtendedWU(IWorkUnit * wu)
 }
 
 
-extern WORKUNIT_API void addExceptionToWorkunit(IWorkUnit * wu, WUExceptionSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column)
+extern WORKUNIT_API void addExceptionToWorkunit(IWorkUnit * wu, ErrorSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column)
 {
     Owned<IWUException> we = wu->createException();
     we->setSeverity(severity);

common/workunit/workunit.hpp

@@ -162,17 +162,6 @@ enum WUResultStatus
 
 
 
-enum WUExceptionSeverity
-{
-    ExceptionSeverityInformation = 0,
-    ExceptionSeverityWarning = 1,
-    ExceptionSeverityError = 2,
-    ExceptionSeverityAlert = 3,
-    ExceptionSeveritySize = 4
-};
-
-
-
 //! IConstWUGraph
 
 enum WUGraphType
@@ -508,7 +497,7 @@ interface IConstWUException : extends IInterface
     virtual IStringVal & getExceptionSource(IStringVal & str) const = 0;
     virtual IStringVal & getExceptionMessage(IStringVal & str) const = 0;
     virtual unsigned getExceptionCode() const = 0;
-    virtual WUExceptionSeverity getSeverity() const = 0;
+    virtual ErrorSeverity getSeverity() const = 0;
     virtual IStringVal & getTimeStamp(IStringVal & dt) const = 0;
     virtual IStringVal & getExceptionFileName(IStringVal & str) const = 0;
     virtual unsigned getExceptionLineNo() const = 0;
@@ -521,7 +510,7 @@ interface IWUException : extends IConstWUException
     virtual void setExceptionSource(const char * str) = 0;
     virtual void setExceptionMessage(const char * str) = 0;
     virtual void setExceptionCode(unsigned code) = 0;
-    virtual void setSeverity(WUExceptionSeverity level) = 0;
+    virtual void setSeverity(ErrorSeverity level) = 0;
     virtual void setTimeStamp(const char * dt) = 0;
     virtual void setExceptionFileName(const char * str) = 0;
     virtual void setExceptionLineNo(unsigned r) = 0;
@@ -1366,7 +1355,7 @@ extern WORKUNIT_API StringBuffer &formatGraphTimerLabel(StringBuffer &str, const
 extern WORKUNIT_API StringBuffer &formatGraphTimerScope(StringBuffer &str, const char *graphName, unsigned subGraphNum, unsigned __int64 subId);
 extern WORKUNIT_API bool parseGraphTimerLabel(const char *label, StringAttr &graphName, unsigned & graphNum, unsigned &subGraphNum, unsigned &subId);
 extern WORKUNIT_API bool parseGraphScope(const char *scope, StringAttr &graphName, unsigned & graphNum, unsigned &subGraphId);
-extern WORKUNIT_API void addExceptionToWorkunit(IWorkUnit * wu, WUExceptionSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column);
+extern WORKUNIT_API void addExceptionToWorkunit(IWorkUnit * wu, ErrorSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column);
 extern WORKUNIT_API void setWorkUnitFactory(IWorkUnitFactory *_factory);
 extern WORKUNIT_API IWorkUnitFactory * getWorkUnitFactory();
 extern WORKUNIT_API IWorkUnitFactory * getWorkUnitFactory(ISecManager *secmgr, ISecUser *secuser);

common/wuwebview/wuwebview.cpp

@@ -71,7 +71,7 @@ public:
     void appendResults(IConstWorkUnit *wu, const char *username, const char *pw)
     {
         StringBufferAdaptor resultXML(buffer);
-        getFullWorkUnitResultsXML(username, pw, wu, resultXML, WorkUnitXML_NoRoot, ExceptionSeverityError);
+        getFullWorkUnitResultsXML(username, pw, wu, resultXML, WorkUnitXML_NoRoot, SeverityError);
     }
 
     void appendSingleResult(IConstWorkUnit *wu, const char *resultname, const char *username, const char *pw)
@@ -665,7 +665,7 @@ void WuWebView::renderResultsJSON(StringBuffer &out, const char *jsonp)
     responseName.append("Response");
     appendJSONName(out, responseName);
     StringBufferAdaptor json(out);
-    getFullWorkUnitResultsJSON(username, pw, cw, json, 0, ExceptionSeverityError);
+    getFullWorkUnitResultsJSON(username, pw, cw, json, 0, SeverityError);
     out.append("}");
     if (jsonp && *jsonp)
         out.append(");");

dali/base/dadfs.cpp

@@ -47,6 +47,7 @@
 #define SDS_CONNECT_TIMEOUT  (1000*60*60*2)     // better than infinite
 #define SDS_SUB_LOCK_TIMEOUT (10000)
 #define SDS_TRANSACTION_RETRY (60000)
+#define SDS_UPDATEFS_TIMEOUT (10000)
 
 #define DEFAULT_NUM_DFS_THREADS 30
 #define TIMEOUT_ON_CLOSEDOWN 120000 // On closedown, give up on trying to join a thread in CDaliDFSServer after two minutes
@@ -986,6 +987,7 @@ public:
         defaultudesc.setown(createUserDescriptor());
         redirection.setown(createDFSredirection());
     }
+    unsigned queryDefaultTimeout() const { return defaultTimeout; }
 
     IDistributedFile *dolookup(CDfsLogicalFileName &logicalname, IUserDescriptor *user, bool writeattr, bool hold, bool lockSuperOwner, IDistributedFileTransaction *transaction, unsigned timeout);
 
@@ -3215,7 +3217,7 @@ protected:
             root.setown(closeConnection(removeFile));
             // NB: The file is now unlocked
             if (removeFile && !logicalName.isExternal())
-                updateFS(logicalName, timeoutMs);
+                updateFS(logicalName, parent->queryDefaultTimeout());
 
             logicalName.clear();
         }
@@ -5486,7 +5488,7 @@ public:
         clearSuperOwners(timeoutMs);
         writeLock.clear();
         root.setown(closeConnection(true));
-        updateFS(logicalName, timeoutMs);
+        updateFS(logicalName, parent->queryDefaultTimeout());
         logicalName.clear();
     }
 

ecl/eclagent/agentctx.hpp

@@ -17,11 +17,23 @@
 #ifndef AGENTCTX_HPP_INCL
 #define AGENTCTX_HPP_INCL
 
+#include "errorlist.h"
 #include "dautils.hpp"
 #include "eclhelper.hpp"
 #include "workunit.hpp"
 #include "layouttrans.hpp"
 
+#if (ECLAGENT_ERROR_START != 5400 || ECLAGENT_ERROR_END != 5499)
+#error "ECLAGENT_ERROR_START has changed"
+#endif
+
+#define WRN_SkipMissingOptIndex             5400
+#define WRN_SkipMissingOptFile              5401
+#define WRN_UseLayoutTranslation            5402
+#define WRN_UnsupportedAlgorithm            5403
+#define WRN_MismatchGroupInfo               5404
+#define WRN_MismatchCompressInfo            5405
+
 struct IHThorGraphResult : extends IInterface
 {
     virtual void addRowOwn(const void * row) = 0;

ecl/eclagent/eclagent.cpp

@@ -1467,12 +1467,14 @@ char * EclAgent::getExpandLogicalName(const char * logicalName)
 
 void EclAgent::addWuException(const char * text, unsigned code, unsigned severity, char const * source)
 {
-    addException((WUExceptionSeverity)severity, source, code, text, NULL, 0, 0, false, false);
+    ErrorSeverity mappedSeverity = wuRead->getWarningSeverity(code, (ErrorSeverity)severity);
+    if (mappedSeverity != SeverityIgnore)
+        addException(mappedSeverity, source, code, text, NULL, 0, 0, false, false);
 }
 
 void EclAgent::addWuAssertFailure(unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool isAbort)
 {
-    addException(ExceptionSeverityError, "user", code, text, filename, lineno, column, false, false);
+    addException(SeverityError, "user", code, text, filename, lineno, column, false, false);
     if (isAbort)
         rtlFailOnAssert();      // minimal implementation
 }
@@ -1639,10 +1641,10 @@ IHThorGraphResults * EclAgent::createGraphLoopResults()
 
 //---------------------------------------------------------------------------
 
-void addException(IWorkUnit *w, WUExceptionSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError)
+void addException(IWorkUnit *w, ErrorSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError)
 {
     PrintLog("%s", text);
-    if ((severity == ExceptionSeverityError) && (w->getState()!=WUStateAborting) && failOnError)
+    if ((severity == SeverityError) && (w->getState()!=WUStateAborting) && failOnError)
         w->setState(WUStateFailed);
     addExceptionToWorkunit(w, severity, source, code, text, filename, lineno, column);
 }
@@ -1953,7 +1955,7 @@ void EclAgent::doProcess()
                     int code = e->errorCode();
                     VStringBuffer msg("Failed to deschedule workunit %s: ", w->queryWuid());
                     e->errorMessage(msg);
-                    logException(ExceptionSeverityWarning, code, msg.str(), false);
+                    logException(SeverityWarning, code, msg.str(), false);
                     e->Release();
                     WARNLOG("%s (%d)", msg.str(), code);
                 }
@@ -2000,7 +2002,7 @@ void EclAgent::doProcess()
             StringBuffer m("System error ");
             m.append(e->errorCode()).append(": ");
             e->errorMessage(m);
-            ::addException(w, ExceptionSeverityError, "eclagent", e->errorCode(), m.str(), NULL, 0, 0, true);
+            ::addException(w, SeverityError, "eclagent", e->errorCode(), m.str(), NULL, 0, 0, true);
         }
         catch (IException *e2)
         {
@@ -2198,7 +2200,7 @@ void EclAgentWorkflowMachine::reportContingencyFailure(char const * type, IExcep
     StringBuffer msg;
     msg.append(type).append(" clause failed (execution will continue): ").append(e->errorCode()).append(": ");
     e->errorMessage(msg);
-    agent.logException(ExceptionSeverityWarning, e->errorCode(), msg.str(), false);
+    agent.logException(SeverityWarning, e->errorCode(), msg.str(), false);
 }
 
 void EclAgentWorkflowMachine::checkForAbort(unsigned wfid, IException * handling)
@@ -2211,7 +2213,7 @@ void EclAgentWorkflowMachine::checkForAbort(unsigned wfid, IException * handling
             msg.append("Abort takes precedence over error: ").append(handling->errorCode()).append(": ");
             handling->errorMessage(msg);
             msg.append(" (in item ").append(wfid).append(")");
-            agent.logException(ExceptionSeverityWarning, handling->errorCode(), msg.str(), false);
+            agent.logException(SeverityWarning, handling->errorCode(), msg.str(), false);
             handling->Release();
         }
         throw new WorkflowException(0, "Workunit abort request received", wfid, WorkflowException::ABORT, MSGAUD_user);
@@ -2308,16 +2310,16 @@ void EclAgent::doNotify(char const * name, char const * text, const char * targe
     pusher->push(name, text, target);
 }
 
-void EclAgent::logException(WUExceptionSeverity severity, unsigned code, const char * text, bool isAbort)
+void EclAgent::logException(ErrorSeverity severity, unsigned code, const char * text, bool isAbort)
 {
     addException(severity, "eclagent", code, text, NULL, 0, 0, true, isAbort);
-    if (severity == ExceptionSeverityError)
+    if (severity == SeverityError)
         ERRLOG(code, "%s", text);
 }
 
-void EclAgent::addException(WUExceptionSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError, bool isAbort)
+void EclAgent::addException(ErrorSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError, bool isAbort)
 {
-    if (writeResultsToStdout && (severity != ExceptionSeverityInformation))
+    if (writeResultsToStdout && (severity != SeverityInformation))
     {
         StringBuffer location;
         if (filename)
@@ -2327,7 +2329,7 @@ void EclAgent::addException(WUExceptionSeverity severity, const char * source, u
                 location.append('(').append(lineno).append(")");
             location.append(": ");
         }
-        const char * kind = (severity == ExceptionSeverityError) ? "error" : "warning";
+        const char * kind = (severity == SeverityError) ? "error" : "warning";
         fprintf(stderr, "%s%s: C%04u %s\n", location.str(), kind, code, text);
     }
     try
@@ -2387,7 +2389,7 @@ void EclAgent::logException(WorkflowException *e)
     else    
         m.append("Unknown error");
 
-    logException(ExceptionSeverityError, code, m.str(), isAbort);
+    logException(SeverityError, code, m.str(), isAbort);
 }
 
 void EclAgent::logException(IException *e)
@@ -2407,7 +2409,7 @@ void EclAgent::logException(IException *e)
     else    
         m.append("Unknown error");
 
-    logException(ExceptionSeverityError, code, m.str(), false);
+    logException(SeverityError, code, m.str(), false);
 }
 
 void EclAgent::logException(std::exception & e)
@@ -2417,7 +2419,7 @@ void EclAgent::logException(std::exception & e)
         m.append("out of memory (std::bad_alloc)");
     else
         m.append("standard library exception (std::exception ").append(e.what()).append(")");
-    logException(ExceptionSeverityError, 0, m.str(), false);
+    logException(SeverityError, 0, m.str(), false);
 }
 
 static unsigned __int64 crcLogicalFileTime(IDistributedFile * file, unsigned __int64 crc, const char * filename)
@@ -2603,7 +2605,7 @@ bool EclAgent::isPersistUptoDate(Owned<IRemoteConnection> &persistLock, IRuntime
         {
             StringBuffer msg;
             msg.append("PERSIST('").append(logicalName).append("') is up to date");
-            logException(ExceptionSeverityInformation, 0, msg.str(), false);
+            logException(SeverityInformation, 0, msg.str(), false);
             return true;
         }
 
@@ -2626,12 +2628,12 @@ bool EclAgent::isPersistUptoDate(Owned<IRemoteConnection> &persistLock, IRuntime
     {
         StringBuffer msg;
         msg.append("PERSIST('").append(logicalName).append("') is up to date (after being calculated by another job)");
-        logException(ExceptionSeverityInformation, 0, msg.str(), false);
+        logException(SeverityInformation, 0, msg.str(), false);
         changePersistLockMode(persistLock, RTM_LOCK_READ, logicalName, true);
         return true;
     }
     if (errText.length())
-        logException(ExceptionSeverityInformation, 0, errText.str(), false);
+        logException(SeverityInformation, 0, errText.str(), false);
     return false;
 }
 
@@ -2685,7 +2687,7 @@ void EclAgent::checkPersistMatches(const char * logicalName, unsigned eclCRC)
 
     StringBuffer msg;
     msg.append("Frozen PERSIST('").append(logicalName).append("') is up to date");
-    logException(ExceptionSeverityInformation, 0, msg.str(), false);
+    logException(SeverityInformation, 0, msg.str(), false);
 }
 
 static int comparePersistAccess(IInterface * const *_a, IInterface * const *_b)
@@ -3068,7 +3070,7 @@ void EclAgent::fatalAbort(bool userabort,const char *excepttext)
         if (userabort) 
             w->setState(WUStateAborted);
         if (excepttext&&*excepttext)
-            addException(ExceptionSeverityError, "ECLAGENT", 1000, excepttext, NULL, 0, 0, true, false);
+            addException(SeverityError, "ECLAGENT", 1000, excepttext, NULL, 0, 0, true, false);
         w->deleteTempFiles(NULL, false, true);
         wuRead.clear(); 
         w->commit();        // needed because we can't unlock the workunit in this thread

ecl/eclagent/eclagent.ipp

@@ -542,11 +542,11 @@ public:
     virtual const char *queryTemporaryFile(const char *fname);
     virtual void deleteFile(const char * logicalName);
 
-    void addException(WUExceptionSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError, bool isAbort);
+    void addException(ErrorSeverity severity, const char * source, unsigned code, const char * text, const char * filename, unsigned lineno, unsigned column, bool failOnError, bool isAbort);
     void logException(IException *e);  
     void logException(WorkflowException *e);  
     void logException(std::exception & e);
-    void logException(WUExceptionSeverity severity, unsigned code, const char * text, bool isAbort);
+    void logException(ErrorSeverity severity, unsigned code, const char * text, bool isAbort);
 
     void doProcess();
     void runProcess(IEclProcess *process);

ecl/eclccserver/eclccserver.cpp

@@ -231,18 +231,18 @@ class EclccCompileThread : public CInterface, implements IPooledThread, implemen
                     err->setExceptionLineNo(atoi(line));
                     err->setExceptionColumn(atoi(col));
                     if (stricmp(errClass, "info")==0)
-                        err->setSeverity(ExceptionSeverityInformation);
+                        err->setSeverity(SeverityInformation);
                     else if (stricmp(errClass, "warning")==0)
-                        err->setSeverity(ExceptionSeverityWarning);
+                        err->setSeverity(SeverityWarning);
                     else
-                        err->setSeverity(ExceptionSeverityError);
+                        err->setSeverity(SeverityError);
                     err->setExceptionCode(atoi(errCode));
                     err->setExceptionMessage(errText);
                     err->setExceptionFileName(file); // any point if it just says stdin?
                 }
                 else
                 {
-                    err->setSeverity(retcode ? ExceptionSeverityError : ExceptionSeverityWarning);
+                    err->setSeverity(retcode ? SeverityError : SeverityWarning);
                     err->setExceptionMessage(errStr);
                     DBGLOG("%s", errStr);
                 }
@@ -488,7 +488,7 @@ public:
         {
             StringBuffer msg;
             e->errorMessage(msg);
-            addExceptionToWorkunit(workunit, ExceptionSeverityError, "eclccserver", e->errorCode(), msg.str(), NULL, 0, 0);
+            addExceptionToWorkunit(workunit, SeverityError, "eclccserver", e->errorCode(), msg.str(), NULL, 0, 0);
             e->Release();
         }
         if (ok)

ecl/hql/hqlerror.cpp

@@ -32,7 +32,7 @@ ErrorSeverity getSeverity(IAtom * name)
     if (name == ignoreAtom)
         return SeverityIgnore;
     if (name == logAtom)
-        return SeverityInfo;
+        return SeverityInformation;
     return SeverityUnknown;
 }
 
@@ -41,7 +41,7 @@ ErrorSeverity queryDefaultSeverity(WarnErrorCategory category)
     if (category == CategoryError)
         return SeverityFatal;
     if (category == CategoryInformation)
-        return SeverityInfo;
+        return SeverityInformation;
     if (category == CategoryMistake)
         return SeverityError;
     return SeverityWarning;
@@ -184,7 +184,7 @@ public:
         {
         case SeverityIgnore:
             return;
-        case SeverityInfo:
+        case SeverityInformation:
             severityText = "info";
             break;
         case SeverityWarning:
@@ -357,14 +357,14 @@ void checkEclVersionCompatible(Shared<IErrorReceiver> & errors, const char * ecl
             else if (minor != LANGUAGE_VERSION_MINOR)
             {
                 VStringBuffer msg("Mismatch in minor version number (%s v %s)", eclVersion, LANGUAGE_VERSION);
-                Owned<IError> warning = createError(CategoryUnexpected, SeverityInfo, HQLERR_VersionMismatch, msg.str(), NULL, 0, 0);
+                Owned<IError> warning = createError(CategoryUnexpected, SeverityInformation, HQLERR_VersionMismatch, msg.str(), NULL, 0, 0);
                 errors.setown(new ErrorInserter(*errors, warning));
             }
             else if (subminor != LANGUAGE_VERSION_SUB)
             {
                 //This adds the warning if any other warnings occur.
                 VStringBuffer msg("Mismatch in subminor version number (%s v %s)", eclVersion, LANGUAGE_VERSION);
-                Owned<IError> warning = createError(CategoryUnexpected, SeverityInfo, HQLERR_VersionMismatch, msg.str(), NULL, 0, 0);
+                Owned<IError> warning = createError(CategoryUnexpected, SeverityInformation, HQLERR_VersionMismatch, msg.str(), NULL, 0, 0);
                 errors.setown(new ErrorInserter(*errors, warning));
             }
         }

ecl/hql/hqlerrors.hpp

@@ -18,14 +18,7 @@
 #define _HQLERRORS_HPP_
 
 
-/* Error severity: obsoleted */
-
-#define ERR_SEMANTIC                1    /* no resulting expression tree */
-#define ERR_SYNTACTIC               2    /* there is an expression tree, but should not be executed */
-#define ERR_FATAL                   3    /* fatal error: unrecoverable error */
-
 ///////////////////////////////////////////////////////////////////////////////
-/* Warning numbers */
 
 #define WRN_MACROEXPANSION          1001 /* error in macro (see error following this warning) */
 #define WRN_LOCALNONDIST            1004 /* LOCAL specified on dataset that is not distributed */
@@ -75,7 +68,8 @@
 #define WRN_SILLY_EXISTS            1051
 #define WRN_INT_OR_RANGE_EXPECTED   1052 /* Integer or integer range (i.e. 2..3) expected when real detected */
 #define WRN_UNRESOLVED_SYMBOL       1053
-//#define ECL_WARN_END          1100
+
+//Do not define any warnings > 1099 - use the range below instead
 
 ///////////////////////////////////////////////////////////////////////////////
 /* Error numbers */

ecl/hql/hqlexpr.cpp

@@ -12573,7 +12573,7 @@ extern IHqlExpression * createCompound(node_operator op, const HqlExprArray & ac
     if (op == no_compound)
         return createCompound(actions);
     else
-        return createActionList(actions);
+        return createActionList(op, actions);
 }
 
 extern IHqlExpression * createComma(IHqlExpression * expr1, IHqlExpression * expr2)

ecl/hql/hqlmeta.cpp

@@ -1558,7 +1558,7 @@ IHqlExpression * getSubSort(IHqlExpression * dataset, IHqlExpression * order, bo
 
 //--------------------------------------------------------------------------------------------------------------------
 
-IHqlExpression * ensureSorted(IHqlExpression * dataset, IHqlExpression * order, bool isLocal, bool ignoreGrouping, bool alwaysLocal, bool allowSubSort, bool requestSpilling)
+IHqlExpression * ensureSorted(IHqlExpression * dataset, IHqlExpression * order, IHqlExpression * parentExpr, bool isLocal, bool ignoreGrouping, bool alwaysLocal, bool allowSubSort, bool requestSpilling)
 {
     if (isAlreadySorted(dataset, order, isLocal||alwaysLocal, ignoreGrouping))
         return LINK(dataset);
@@ -1572,10 +1572,17 @@ IHqlExpression * ensureSorted(IHqlExpression * dataset, IHqlExpression * order,
                 return subsorted.getClear();
         }
     }
-
-    IHqlExpression * attr1 = isLocal ? createLocalAttribute() : (isGrouped(dataset) && ignoreGrouping) ? createAttribute(globalAtom) : NULL;
-    IHqlExpression * attr2 = requestSpilling ? createAttribute(spillAtom) : NULL;
-    return createDatasetF(no_sort, LINK(dataset), LINK(order), createComma(attr1, attr2), NULL);
+    HqlExprArray args;
+    args.append(OLINK(*dataset));
+    args.append(OLINK(*order));
+    if (isLocal)
+        args.append(*createLocalAttribute());
+    else if (isGrouped(dataset) && ignoreGrouping)
+        args.append(*createAttribute(globalAtom));
+    if (requestSpilling)
+        args.append(*createAttribute(spillAtom));
+    unwindHintAttrs(args, parentExpr);
+    return createDataset(no_sort, args);
 }
 
 //-------------------------------

ecl/hql/hqlmeta.hpp

@@ -110,7 +110,7 @@ extern HQL_API bool matchesAnyDistribution(IHqlExpression * distn);
 extern HQL_API bool appearsToBeSorted(IHqlExpression * dataset, bool isLocal, bool ignoreGrouping);
 extern HQL_API bool isAlreadySorted(IHqlExpression * dataset, const HqlExprArray & newSort, bool isLocal, bool ignoreGrouping);
 extern HQL_API bool isAlreadySorted(IHqlExpression * dataset, IHqlExpression * newSort, bool isLocal, bool ignoreGrouping);
-extern HQL_API IHqlExpression * ensureSorted(IHqlExpression * dataset, IHqlExpression * order, bool isLocal, bool ignoreGrouping, bool alwaysLocal, bool allowSubSort, bool requestSpilling);
+extern HQL_API IHqlExpression * ensureSorted(IHqlExpression * dataset, IHqlExpression * order, IHqlExpression * parentExpr, bool isLocal, bool ignoreGrouping, bool alwaysLocal, bool allowSubSort, bool requestSpilling);
 
 extern HQL_API bool isWorthShuffling(IHqlExpression * dataset, IHqlExpression * order, bool isLocal, bool ignoreGrouping);
 extern HQL_API bool isWorthShuffling(IHqlExpression * dataset, const HqlExprArray & newSort, bool isLocal, bool ignoreGrouping);

ecl/hql/hqlwuerr.cpp

@@ -50,21 +50,21 @@ IError * WorkUnitErrorReceiver::mapError(IError * error)
 
 void WorkUnitErrorReceiver::report(IError* eclError)
 {
-    WUExceptionSeverity wuSeverity = ExceptionSeverityInformation;
+    ErrorSeverity wuSeverity = SeverityInformation;
     ErrorSeverity severity = eclError->getSeverity();
 
     switch (severity)
     {
     case SeverityIgnore:
         return;
-    case SeverityInfo:
+    case SeverityInformation:
         break;
     case SeverityWarning:
-        wuSeverity = ExceptionSeverityWarning;
+        wuSeverity = SeverityWarning;
         break;
     case SeverityError:
     case SeverityFatal:
-        wuSeverity = ExceptionSeverityError;
+        wuSeverity = SeverityError;
         break;
     }
 
@@ -81,7 +81,7 @@ size32_t WorkUnitErrorReceiver::errCount()
     unsigned count = 0;
     Owned<IConstWUExceptionIterator> exceptions = &wu->getExceptions();
     ForEach(*exceptions)
-        if (exceptions->query().getSeverity() == ExceptionSeverityError)
+        if (exceptions->query().getSeverity() == SeverityError)
             count++;
     return count;
 }
@@ -91,7 +91,7 @@ size32_t WorkUnitErrorReceiver::warnCount()
     unsigned count = 0;
     Owned<IConstWUExceptionIterator> exceptions = &wu->getExceptions();
     ForEach(*exceptions)
-        if (exceptions->query().getSeverity() == ExceptionSeverityWarning)
+        if (exceptions->query().getSeverity() == SeverityWarning)
             count++;
     return count;
 }

ecl/hqlcpp/hqlcpp.cpp

@@ -2078,7 +2078,7 @@ void HqlCppTranslator::reportWarning(WarnErrorCategory category, unsigned id, co
     doReportWarning(category, SeverityUnknown, NULL, id, s.str());
 }
 
-void HqlCppTranslator::addWorkunitException(WUExceptionSeverity severity, unsigned code, const char * text, IHqlExpression * location)
+void HqlCppTranslator::addWorkunitException(ErrorSeverity severity, unsigned code, const char * text, IHqlExpression * location)
 {
     Owned<IWUException> msg = wu()->createException();
     msg->setExceptionSource("Code Generator");

ecl/hqlcpp/hqlcpp.ipp

@@ -924,7 +924,7 @@ public:
     void reportWarning(WarnErrorCategory category, ErrorSeverity explicitSeverity, IHqlExpression * location, unsigned id, const char * msg, ...) __attribute__((format(printf, 6, 7)));
     void reportError(IHqlExpression * location, int code, const char *format, ...) __attribute__((format(printf, 4, 5)));
     void reportErrorDirect(IHqlExpression * location, int code,const char *msg, bool alwaysAbort);
-    void addWorkunitException(WUExceptionSeverity severity, unsigned code, const char * msg, IHqlExpression * location);
+    void addWorkunitException(ErrorSeverity severity, unsigned code, const char * msg, IHqlExpression * location);
     void useFunction(IHqlExpression * funcdef);
     void useLibrary(const char * libname);
     void finalizeResources();

ecl/hqlcpp/hqlcset.cpp

@@ -817,6 +817,16 @@ InlineLinkedDictionaryCursor::InlineLinkedDictionaryCursor(HqlCppTranslator & _t
 {
 }
 
+IHqlExpression * InlineLinkedDictionaryCursor::getFirstSearchValue(IHqlExpression * searchExpr, IHqlExpression * searchRecord)
+{
+    if (searchExpr->getOperator() == no_alias)
+        searchExpr = searchExpr->queryChild(0);
+
+    IHqlExpression * matched = getExtractSelect(searchExpr->queryChild(0), queryFirstField(searchRecord), false);
+    assertex(matched);
+    return matched;
+}
+
 BoundRow * InlineLinkedDictionaryCursor::buildSelectMap(BuildCtx & ctx, IHqlExpression * mapExpr)
 {
     Owned<BoundRow> tempRow = translator.declareLinkedRow(ctx, mapExpr, false);
@@ -863,7 +873,7 @@ BoundRow * InlineLinkedDictionaryCursor::buildSelectMap(BuildCtx & ctx, IHqlExpr
     if (optimizedLookupFunc.length())
     {
         args.add(*LINK(dictionary), 0);
-        args.append(*getExtractSelect(searchExpr->queryChild(0), queryFirstField(searchRecord), false));
+        args.append(*getFirstSearchValue(searchExpr, searchRecord));
         args.append(*::createRow(no_null, LINK(record))); // the default record
         lookupFunction = createIdAtom(optimizedLookupFunc);
     }
@@ -889,6 +899,7 @@ void InlineLinkedDictionaryCursor::buildInDataset(BuildCtx & ctx, IHqlExpression
     IHqlExpression *record = ds->queryRecord();
     IHqlExpression *dictionary = inExpr->queryChild(1);
     IHqlExpression *searchExpr = inExpr->queryChild(0);
+
     HqlExprArray args;
 
     OwnedHqlExpr searchRecord = getDictionarySearchRecord(record);
@@ -927,7 +938,7 @@ void InlineLinkedDictionaryCursor::buildInDataset(BuildCtx & ctx, IHqlExpression
     if (optimizedLookupFunc.length())
     {
         args.add(*LINK(dictionary), 0);
-        args.append(*getExtractSelect(searchExpr->queryChild(0), queryFirstField(searchRecord), false));
+        args.append(*getFirstSearchValue(searchExpr, searchRecord));
         lookupFunction = createIdAtom(optimizedLookupFunc);
     }
     else

ecl/hqlcpp/hqlcset.ipp

@@ -108,6 +108,9 @@ public:
     virtual void buildIterateClass(BuildCtx & ctx, StringBuffer & cursorName, BuildCtx * initctx) { throwUnexpected(); }
     virtual void buildCountDict(BuildCtx & ctx, CHqlBoundExpr & tgt);
     virtual void buildExistsDict(BuildCtx & ctx, CHqlBoundExpr & tgt);
+
+private:
+    IHqlExpression * getFirstSearchValue(IHqlExpression * searchExpr, IHqlExpression * searchRecord);
 };
 
 class MultiLevelDatasetCursor : public BaseDatasetCursor

ecl/hqlcpp/hqliproj.cpp

@@ -2699,7 +2699,7 @@ void ImplicitProjectTransformer::logChange(const char * message, IHqlExpression
         {
             StringBuffer messageText;
             messageText.appendf(format, message, name.str(), fieldText.str());
-            translator.addWorkunitException(ExceptionSeverityInformation, 0, messageText.str(), NULL);
+            translator.addWorkunitException(SeverityInformation, 0, messageText.str(), NULL);
         }
     }
 }

ecl/hqlcpp/hqlttcpp.cpp

@@ -1920,7 +1920,7 @@ static IHqlExpression * normalizeIndexBuild(IHqlExpression * expr, bool sortInde
             }
         }
 
-        OwnedHqlExpr sorted = ensureSorted(dataset, newsort, expr->hasAttribute(localAtom), true, alwaysLocal, allowImplicitSubSort, true);
+        OwnedHqlExpr sorted = ensureSorted(dataset, newsort, expr, expr->hasAttribute(localAtom), true, alwaysLocal, allowImplicitSubSort, true);
         if (sorted == dataset)
             return NULL;
 
@@ -2523,7 +2523,7 @@ IHqlExpression * ThorHqlTransformer::normalizeCoGroup(IHqlExpression * expr)
         {
             IHqlExpression & cur = inputs.item(i);
             OwnedHqlExpr mappedOrder = replaceSelector(bestSortOrder, queryActiveTableSelector(), &cur);
-            sortedInputs.append(*ensureSorted(&cur, mappedOrder, true, true, alwaysLocal, options.implicitSubSort, false));
+            sortedInputs.append(*ensureSorted(&cur, mappedOrder, expr, true, true, alwaysLocal, options.implicitSubSort, false));
         }
         HqlExprArray sortedArgs;
         unwindChildren(sortedArgs, bestSortOrder);
@@ -2565,7 +2565,7 @@ static IHqlExpression * getNonThorSortedJoinInput(IHqlExpression * joinExpr, IHq
     groupOrder.setown(replaceSelector(groupOrder, queryActiveTableSelector(), expr->queryNormalizedSelector()));
 
     //not used for thor, so sort can be local
-    OwnedHqlExpr table = ensureSorted(expr, groupOrder, false, true, true, implicitSubSort, false);
+    OwnedHqlExpr table = ensureSorted(expr, groupOrder, joinExpr, false, true, true, implicitSubSort, false);
     if (table != expr)
         table.setown(cloneInheritedAnnotations(joinExpr, table));
 
@@ -6023,7 +6023,7 @@ IHqlExpression * WorkflowTransformer::extractCommonWorkflow(IHqlExpression * exp
             s.append("[").append(expr->queryName()).append("] ");
         s.append(" to common up code between workflow items");
         DBGLOG("%s", s.str());
-        translator.addWorkunitException(ExceptionSeverityInformation, HQLWRN_TryAddingIndependent, s.str(), location);
+        translator.addWorkunitException(SeverityInformation, HQLWRN_TryAddingIndependent, s.str(), location);
         if (!translator.queryOptions().performWorkflowCse)
             return LINK(transformed);
     }
@@ -6037,7 +6037,7 @@ IHqlExpression * WorkflowTransformer::extractCommonWorkflow(IHqlExpression * exp
         s.append("[").append(expr->queryName()).append("] ");
     s.append(" to common up between workflow items [").append(wfid).append("]");
     DBGLOG("%s", s.str());
-    translator.addWorkunitException(ExceptionSeverityInformation, 0, s.str(), location);
+    translator.addWorkunitException(SeverityInformation, 0, s.str(), location);
 
     GlobalAttributeInfo info("spill::wfa", "wfa", transformed);
     info.extractGlobal(NULL, translator.getTargetClusterType());       // should really be a slightly different function

ecl/hthor/hthor.cpp

@@ -3843,7 +3843,7 @@ void CHThorGroupSortActivity::createSorter()
     {
         StringBuffer sb;
         sb.appendf("Ignoring unsupported sort order algorithm '%s', using default", algoname.get());
-        agent.addWuException(sb.str(),0,ExceptionSeverityWarning,"hthor");
+        agent.addWuException(sb.str(),WRN_UnsupportedAlgorithm,SeverityWarning,"hthor");
         if((flags & TAFunstable) != 0)
             sorter.setown(new CQuickSorter(helper.queryCompare(), queryRowManager(), InitialSortElements, CommitStep));
         else
@@ -6128,16 +6128,6 @@ CHThorRemoteResultActivity::CHThorRemoteResultActivity(IAgentContext &_agent, un
 void CHThorRemoteResultActivity::execute()
 {
     OwnedConstRoxieRow result(input->nextInGroup());
-#if 0
-    //This isn't correct - it can have a null input - at least until TAKaction is generated for that case.
-    if(!result)
-    {
-        char const * msg = "Remote Result activity had null input, not sending result";
-        WARNLOG(msg);
-        agent.addWuException(msg, 0, ExceptionSeverityWarning, "hthor");
-        return;
-    }
-#endif
     helper.sendResult(result);
 }
 
@@ -7861,7 +7851,7 @@ void CHThorDiskReadBaseActivity::resolve()
             StringBuffer buff;
             buff.appendf("Input file '%s' was missing but declared optional", mangledHelperFileName.str());
             WARNLOG("%s", buff.str());
-            agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+            agent.addWuException(buff.str(), WRN_SkipMissingOptFile, SeverityInformation, "hthor");
         }
     }
 }
@@ -7878,7 +7868,7 @@ void CHThorDiskReadBaseActivity::gatherInfo(IFileDescriptor * fileDesc)
                 StringBuffer msg;
                 msg.append("DFS and code generated group info. differs: DFS(").append(grouped ? "grouped" : "ungrouped").append("), CodeGen(").append(grouped ? "ungrouped" : "grouped").append("), using DFS info");
                 WARNLOG("%s", msg.str());
-                agent.addWuException(msg.str(), 0, ExceptionSeverityWarning, "hthor");
+                agent.addWuException(msg.str(), WRN_MismatchGroupInfo, SeverityError, "hthor");
             }
         }
         else
@@ -7915,7 +7905,7 @@ void CHThorDiskReadBaseActivity::gatherInfo(IFileDescriptor * fileDesc)
                 StringBuffer msg;
                 msg.append("Ignoring compression attribute on file ").append(mangledHelperFileName.str()).append(", which is not published as compressed");
                 WARNLOG("%s", msg.str());
-                agent.addWuException(msg.str(), 0, ExceptionSeverityWarning, "hthor");
+                agent.addWuException(msg.str(), WRN_MismatchCompressInfo, SeverityWarning, "hthor");
                 compressed = true;
             }
         }

ecl/hthor/hthorkey.cpp

@@ -701,7 +701,7 @@ void CHThorIndexReadActivityBase::getLayoutTranslators()
                 StringBuffer buff;
                 buff.append("Using record layout translation to correct layout mismatch on reading index ").append(f.queryLogicalName());
                 WARNLOG("%s", buff.str());
-                agent.addWuException(buff.str(), 0, ExceptionSeverityWarning, "hthor");
+                agent.addWuException(buff.str(), WRN_UseLayoutTranslation, SeverityWarning, "hthor");
             }
             layoutTransArray.append(layoutTrans.getClear());
         } while(superIterator->next());
@@ -714,7 +714,7 @@ void CHThorIndexReadActivityBase::getLayoutTranslators()
             StringBuffer buff;
             buff.append("Using record layout translation to correct layout mismatch on reading index ").append(df->queryLogicalName());
             WARNLOG("%s", buff.str());
-            agent.addWuException(buff.str(), 0, ExceptionSeverityWarning, "hthor");
+            agent.addWuException(buff.str(), WRN_UseLayoutTranslation, SeverityWarning, "hthor");
         }
     }
 }
@@ -1080,7 +1080,7 @@ extern HTHOR_API IHThorActivity *createIndexReadActivity(IAgentContext &_agent,
         StringBuffer buff;
         buff.append("Skipping OPT index read of nonexistent file ").append(lfn);
         WARNLOG("%s", buff.str());
-        _agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+        _agent.addWuException(buff.str(), WRN_SkipMissingOptIndex, SeverityInformation, "hthor");
         return new CHThorNullActivity(_agent, _activityId, _subgraphId, arg, _kind);
     }
     _agent.logFileAccess(dFile, "HThor", "READ");
@@ -1262,7 +1262,7 @@ extern HTHOR_API IHThorActivity *createIndexNormalizeActivity(IAgentContext &_ag
         StringBuffer buff;
         buff.append("Skipping OPT index normalize of nonexistent file ").append(lfn);
         WARNLOG("%s", buff.str());
-        _agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+        _agent.addWuException(buff.str(), WRN_SkipMissingOptIndex, SeverityInformation, "hthor");
         return new CHThorNullActivity(_agent, _activityId, _subgraphId, arg, _kind);
     }
     _agent.logFileAccess(dFile, "HThor", "READ");
@@ -1380,7 +1380,7 @@ extern HTHOR_API IHThorActivity *createIndexAggregateActivity(IAgentContext &_ag
         StringBuffer buff;
         buff.append("Skipping OPT index aggregate of nonexistent file ").append(lfn);
         WARNLOG("%s", buff.str());
-        _agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+        _agent.addWuException(buff.str(), WRN_SkipMissingOptIndex, SeverityInformation, "hthor");
         return new CHThorNullAggregateActivity(_agent, _activityId, _subgraphId, arg, arg, _kind);
     }
     _agent.logFileAccess(dFile, "HThor", "READ");
@@ -1486,7 +1486,7 @@ extern HTHOR_API IHThorActivity *createIndexCountActivity(IAgentContext &_agent,
         StringBuffer buff;
         buff.append("Skipping OPT index count of nonexistent file ").append(lfn);
         WARNLOG("%s", buff.str());
-        _agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+        _agent.addWuException(buff.str(), WRN_SkipMissingOptIndex, SeverityInformation, "hthor");
         return new CHThorNullCountActivity(_agent, _activityId, _subgraphId, arg, _kind);
     }
     _agent.logFileAccess(dFile, "HThor", "READ");
@@ -1598,7 +1598,7 @@ extern HTHOR_API IHThorActivity *createIndexGroupAggregateActivity(IAgentContext
         StringBuffer buff;
         buff.append("Skipping OPT index group aggregate of nonexistent file ").append(lfn);
         WARNLOG("%s", buff.str());
-        _agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+        _agent.addWuException(buff.str(), WRN_SkipMissingOptIndex, SeverityInformation, "hthor");
         return new CHThorNullActivity(_agent, _activityId, _subgraphId, arg, _kind);
     }
     _agent.logFileAccess(dFile, "HThor", "READ");
@@ -2217,7 +2217,7 @@ public:
                 StringBuffer buff;
                 buff.append("Skipping OPT fetch of nonexistent file ").append(lfn);
                 WARNLOG("%s", buff.str());
-                agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+                agent.addWuException(buff.str(), WRN_SkipMissingOptFile, SeverityInformation, "hthor");
             }
         }
         inputThread.setown(new InputHandler(this));
@@ -2482,7 +2482,7 @@ public:
             StringBuffer buff;
             buff.append("Skipping OPT fetch of nonexistent file ").append(lfn);
             WARNLOG("%s", buff.str());
-            agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+            agent.addWuException(buff.str(), WRN_SkipMissingOptFile, SeverityInformation, "hthor");
         }
             
         csvSplitter.init(_arg.getMaxColumns(), csvInfo, quotes, separators, terminators, escapes);
@@ -3929,7 +3929,7 @@ public:
             StringBuffer buff;
             buff.append("Skipping OPT keyed join against nonexistent file ").append(lfn);
             WARNLOG("%s", buff.str());
-            agent.addWuException(buff.str(), 0, ExceptionSeverityInformation, "hthor");
+            agent.addWuException(buff.str(), WRN_SkipMissingOptFile, SeverityInformation, "hthor");
         }
         CHThorThreadedActivityBase::start();
     }
@@ -4073,14 +4073,6 @@ protected:
             ::verifyFormatCrcSuper(helper.getDiskFormatCrc(), f, false, true);
     }
 
-    virtual void warn(char const * msg)
-    {
-        StringBuffer buff;
-        buff.append(msg).append(" for index ").append(dFile->queryLogicalName());
-        WARNLOG("%s", buff.str());
-        agent.addWuException(buff.str(), 0, ExceptionSeverityWarning, "hthor");
-    }
-
     virtual void fail(char const * msg)
     {
         throw MakeStringExceptionDirect(0, msg);

esp/build.sh

@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+echoerr() { echo "$@" 1>&2; }
 
 set -e
 
@@ -26,7 +27,7 @@ rm -rf "$DISTDIR"
 echo " Done"
 
 if [ ! -d "$TOOLSDIR" ]; then
-    echo "Can't find Dojo build tools -- did you initialise submodules? (git submodule update --init --recursive)"
+    echoerr "ERROR:  Can't find Dojo build tools -- did you initialise submodules? (git submodule update --init --recursive)"
     exit 1
 fi
 
@@ -41,16 +42,14 @@ perl -pe "
 
 echo "Building: $SRCDIR/Visualization"
 cd "$SRCDIR/Visualization/"
-./build.sh
-mkdir -p "$DISTDIR/Visualization"
-cp -r "$SRCDIR/Visualization/build" "$DISTDIR/Visualization/widgets"
+./build.sh "$DISTDIR/Visualization/widgets"
 
 cd "$TOOLSDIR"
 
 if which node >/dev/null; then
     node ../../dojo/dojo.js baseUrl=../../dojo load=build --profile "$PROFILE" --releaseDir "$DISTDIR" ${*:2}
 else
-    echo "node.js is required to build ECL Watch - see https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager"
+    echoerr "ERROR:  node.js is required to build - see https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager"
     exit 1
 fi
 

esp/scm/ws_fs.ecm

@@ -88,6 +88,12 @@ ESPStruct [nil_remove] DFUWorkunit
     [min_ver("1.10")] bool quotedTerminator(true);
 };
 
+ESPStruct [nil_remove] GroupNode
+{
+    string Name;
+    string ClusterType;
+};
+
 ESPStruct DFUException
 {
     int Code;
@@ -610,6 +616,15 @@ ESPresponse [exceptions_inline] UploadFilesResponse
     ESParray<ESPstruct DFUActionResult> UploadFileResults;
 };
 
+ESPrequest GetSprayTargetsRequest
+{
+};
+
+ESPresponse [exceptions_inline, nil_remove] GetSprayTargetsResponse
+{
+    ESParray<ESPstruct GroupNode, GroupNode> GroupNodes; 
+};
+
 ESPservice [
     version("1.10"), default_client_version("1.10"),
     exceptions_inline("./smc_xslt/exceptions.xslt")] FileSpray
@@ -644,6 +659,7 @@ ESPservice [
     ESPmethod [resp_xsl_default("/esp/xslt/opensave.xslt")] OpenSave(OpenSaveRequest, OpenSaveResponse);
     ESPmethod [resp_xsl_default("/esp/xslt/dropzonefile.xslt")] DropZoneFiles(DropZoneFilesRequest, DropZoneFilesResponse);
     ESPmethod [resp_xsl_default("/esp/xslt/dfuwuaction_results.xslt")] DeleteDropZoneFiles(DeleteDropZoneFilesRequest, DFUWorkunitsActionResponse);
+    ESPmethod GetSprayTargets(GetSprayTargetsRequest, GetSprayTargetsResponse);
 };
 
 SCMexportdef(FileSpray);

esp/services/ecldirect/EclDirectService.cpp

@@ -52,16 +52,16 @@ EclDirectWUExceptions::EclDirectWUExceptions(IConstWorkUnit& cw)
         switch (it->query().getSeverity())
         {
             default:
-            case ExceptionSeverityError:
+            case SeverityError:
                 e->setSeverity("Error");
                 break;
-            case ExceptionSeverityWarning:
+            case SeverityWarning:
                 e->setSeverity("Warning");
                 break;
-            case ExceptionSeverityInformation:
+            case SeverityInformation:
                 e->setSeverity("Info");
                 break;
-            case ExceptionSeverityAlert:
+            case SeverityAlert:
                 e->setSeverity("Alert");
                 break;
         }

esp/services/ws_fs/ws_fsService.cpp

@@ -3182,6 +3182,85 @@ bool CFileSprayEx::onDeleteDropZoneFiles(IEspContext &context, IEspDeleteDropZon
         resp.setDFUActionResults(results);
     }
     catch(IException* e)
+    {
+        FORWARDEXCEPTION(context, e,  ECLWATCH_INTERNAL_ERROR);
+    }
+
+    return true;
+}
+
+void CFileSprayEx::appendGroupNode(IArrayOf<IEspGroupNode>& groupNodes, const char* nodeName, const char* clusterType)
+{
+    Owned<IEspGroupNode> node = createGroupNode();
+    node->setName(nodeName);
+    node->setClusterType(clusterType);
+    groupNodes.append(*node.getClear());
+}
+
+bool CFileSprayEx::onGetSprayTargets(IEspContext &context, IEspGetSprayTargetsRequest &req, IEspGetSprayTargetsResponse &resp)
+{
+    try
+    {
+        if (!context.validateFeatureAccess(FILE_SPRAY_URL, SecAccess_Read, false))
+            throw MakeStringException(ECLWATCH_FILE_SPRAY_ACCESS_DENIED, "Permission denied.");
+
+        Owned<IEnvironmentFactory> factory = getEnvironmentFactory();
+        Owned<IConstEnvironment> environment = factory->openEnvironment();
+        Owned<IPropertyTree> root = &environment->getPTree();
+        if (!root)
+            throw MakeStringException(ECLWATCH_CANNOT_GET_ENV_INFO, "Failed to get environment information.");
+
+        IArrayOf<IEspGroupNode> sprayTargets;
+        //Fetch all the group names for all the thor instances (and dedup them)
+        BoolHash uniqueThorClusterGroupNames;
+        Owned<IPropertyTreeIterator> it = root->getElements("Software/ThorCluster");
+        ForEach(*it)
+        {
+            IPropertyTree& cluster = it->query();
+
+            StringBuffer thorClusterGroupName;
+            getClusterGroupName(cluster, thorClusterGroupName);
+            if (!thorClusterGroupName.length())
+                continue;
+
+            bool* found = uniqueThorClusterGroupNames.getValue(thorClusterGroupName.str());
+            if (!found || !*found)
+                appendGroupNode(sprayTargets, thorClusterGroupName.str(), "thor");
+        }
+
+        //Fetch all the group names for all the hthor instances
+        it.setown(root->getElements("Software/EclAgentProcess"));
+        ForEach(*it)
+        {
+            IPropertyTree &cluster = it->query();
+            const char* name = cluster.queryProp("@name");
+            if (!name || !*name)
+                continue;
+
+            unsigned ins = 0;
+            Owned<IPropertyTreeIterator> insts = cluster.getElements("Instance");
+            ForEach(*insts)
+            {
+                const char *na = insts->query().queryProp("@netAddress");
+                if (!na || !*na)
+                    continue;
+
+                SocketEndpoint ep(na);
+                if (ep.isNull())
+                    continue;
+
+                ins++;
+                VStringBuffer gname("hthor__%s", name);
+                if (ins>1)
+                    gname.append('_').append(ins);
+
+                appendGroupNode(sprayTargets, gname.str(), "hthor");
+            }
+        }
+
+        resp.setGroupNodes(sprayTargets);
+    }
+    catch(IException* e)
     {   
         FORWARDEXCEPTION(context, e,  ECLWATCH_INTERNAL_ERROR);
     }

esp/services/ws_fs/ws_fsService.hpp

@@ -94,6 +94,7 @@ public:
     virtual bool onOpenSave(IEspContext &context, IEspOpenSaveRequest &req, IEspOpenSaveResponse &resp);
     virtual bool onDropZoneFiles(IEspContext &context, IEspDropZoneFilesRequest &req, IEspDropZoneFilesResponse &resp);
     virtual bool onDeleteDropZoneFiles(IEspContext &context, IEspDeleteDropZoneFilesRequest &req, IEspDFUWorkunitsActionResponse &resp);
+    virtual bool onGetSprayTargets(IEspContext &context, IEspGetSprayTargetsRequest &req, IEspGetSprayTargetsResponse &resp);
 
 protected:
     StringBuffer m_QueueLabel;
@@ -111,6 +112,7 @@ protected:
     bool ParseLogicalPath(const char * pLogicalPath, StringBuffer &title);
     bool ParseLogicalPath(const char * pLogicalPath, const char *group, const char* cluster, StringBuffer &folder, StringBuffer &title, StringBuffer &defaultFolder, StringBuffer &defaultReplicateFolder);
     StringBuffer& getAcceptLanguage(IEspContext& context, StringBuffer& acceptLanguage);
+    void appendGroupNode(IArrayOf<IEspGroupNode>& groupNodes, const char* nodeName, const char* clusterType);
 };
 
 #endif //_ESPWIZ_FileSpray_HPP__

esp/services/ws_workunits/ws_workunitsHelpers.cpp

@@ -158,10 +158,10 @@ WsWUExceptions::WsWUExceptions(IConstWorkUnit& wu): numerr(0), numwrn(0), numinf
         switch (it->query().getSeverity())
         {
             default:
-            case ExceptionSeverityError: label = "Error"; numerr++; break;
-            case ExceptionSeverityWarning: label = "Warning"; numwrn++; break;
-            case ExceptionSeverityInformation: label = "Info"; numinf++; break;
-            case ExceptionSeverityAlert: label = "Alert"; numalert++; break;
+            case SeverityError: label = "Error"; numerr++; break;
+            case SeverityWarning: label = "Warning"; numwrn++; break;
+            case SeverityInformation: label = "Info"; numinf++; break;
+            case SeverityAlert: label = "Alert"; numalert++; break;
         }
 
         e->setSeverity(label);
@@ -3033,7 +3033,7 @@ void WsWuHelpers::runWsWorkunit(IEspContext &context, IConstWorkUnit *cw, const
     submitWsWorkunit(context, cw, cluster, NULL, 0, false, true, true, paramXml, variables, debugs);
 }
 
-IException * WsWuHelpers::noteException(IWorkUnit *wu, IException *e, WUExceptionSeverity level)
+IException * WsWuHelpers::noteException(IWorkUnit *wu, IException *e, ErrorSeverity level)
 {
     if (wu)
     {
@@ -3042,7 +3042,7 @@ IException * WsWuHelpers::noteException(IWorkUnit *wu, IException *e, WUExceptio
         we->setExceptionMessage(e->errorMessage(s).str());
         we->setExceptionSource("WsWorkunits");
         we->setSeverity(level);
-        if (level==ExceptionSeverityError)
+        if (level==SeverityError)
             wu->setState(WUStateFailed);
     }
     return e;

esp/services/ws_workunits/ws_workunitsHelpers.hpp

@@ -379,7 +379,7 @@ namespace WsWuHelpers
             IArrayOf<IConstNamedValue> *variables=NULL, IArrayOf<IConstNamedValue> *debugs=NULL);
     void runWsWorkunit(IEspContext &context, IConstWorkUnit *cw, const char *srcWuid, const char *cluster, const char *paramXml=NULL,
             IArrayOf<IConstNamedValue> *variables=NULL, IArrayOf<IConstNamedValue> *debugs=NULL);
-    IException * noteException(IWorkUnit *wu, IException *e, WUExceptionSeverity level=ExceptionSeverityError);
+    IException * noteException(IWorkUnit *wu, IException *e, ErrorSeverity level=SeverityError);
     StringBuffer & resolveQueryWuid(StringBuffer &wuid, const char *queryset, const char *query, bool notSuspended=true, IWorkUnit *wu=NULL);
     void runWsWuQuery(IEspContext &context, IConstWorkUnit *cw, const char *queryset, const char *query, const char *cluster, const char *paramXml=NULL);
     void runWsWuQuery(IEspContext &context, StringBuffer &wuid, const char *queryset, const char *query, const char *cluster, const char *paramXml=NULL);

esp/services/ws_workunits/ws_workunitsService.cpp

@@ -1065,18 +1065,18 @@ bool CWsWorkunitsEx::onWUSubmit(IEspContext &context, IEspWUSubmitRequest &req,
     return true;
 }
 
-WUExceptionSeverity checkGetExceptionSeverity(CWUExceptionSeverity severity)
+ErrorSeverity checkGetExceptionSeverity(CWUExceptionSeverity severity)
 {
     switch (severity)
     {
         case CWUExceptionSeverity_INFO:
-            return ExceptionSeverityInformation;
+            return SeverityInformation;
         case CWUExceptionSeverity_WARNING:
-            return ExceptionSeverityWarning;
+            return SeverityWarning;
         case CWUExceptionSeverity_ERROR:
-            return ExceptionSeverityError;
+            return SeverityError;
         case CWUExceptionSeverity_ALERT:
-            return ExceptionSeverityAlert;
+            return SeverityAlert;
     }
 
     throw MakeStringExceptionDirect(ECLWATCH_INVALID_INPUT,"invalid exception severity");
@@ -1094,7 +1094,7 @@ bool CWsWorkunitsEx::onWURun(IEspContext &context, IEspWURunRequest &req, IEspWU
         const char* runWuid = wuidStr.trim().str();
         StringBuffer wuid;
 
-        WUExceptionSeverity severity = checkGetExceptionSeverity(req.getExceptionSeverity());
+        ErrorSeverity severity = checkGetExceptionSeverity(req.getExceptionSeverity());
 
         if (runWuid && *runWuid)
         {
@@ -3980,16 +3980,16 @@ void CWsWorkunitsEx::createZAPWUInfoFile(IEspWUCreateZAPInfoRequest &req, Owned<
         SCMStringBuffer temp;
         switch (exceptions->query().getSeverity())
         {
-        case ExceptionSeverityInformation:
+        case SeverityInformation:
             info.append("\t").append(exceptions->query().getExceptionMessage(temp)).append("\r\n\r\n");
             break;
-        case ExceptionSeverityWarning:
+        case SeverityWarning:
             warn.append("\t").append(exceptions->query().getExceptionMessage(temp)).append("\r\n\r\n");
             break;
-        case ExceptionSeverityError:
+        case SeverityError:
             err.append("\t").append(exceptions->query().getExceptionMessage(temp)).append("\r\n\r\n");
             break;
-        case ExceptionSeverityAlert:
+        case SeverityAlert:
             alert.append("\t").append(exceptions->query().getExceptionMessage(temp)).append("\r\n\r\n");
             break;
         }

esp/src/Visualization

@@ -1 +1 @@
-Subproject commit f4193ed0c3960452b808a00f6f17e61ca0a51056
+Subproject commit 1074723616023d5e04f03576b390f01b51e57231

esp/src/eclwatch/WsWorkunits.js

@@ -234,12 +234,11 @@ define([
             return ESPRequest.send("WsWorkunits", "WUAction", {
                 request: request,
                 load: function (response) {
-                    arrayUtil.forEach(workunits, function (item, index) {
-                        if (item.refresh) { //  if action is delete then there will be no refresh
-                            item.refresh();
-                        }
-                    });
                     if (lang.exists("WUActionResponse.ActionResults.WUActionResult", response)) {
+                        var wuMap = {};
+                        arrayUtil.forEach(workunits, function (item, index) {
+                            wuMap[item.Wuid] = item;
+                        });
                         arrayUtil.forEach(response.WUActionResponse.ActionResults.WUActionResult, function (item, index) {
                             if (item.Result.indexOf("Failed:") === 0) {
                                 topic.publish("hpcc/brToaster", {
@@ -247,6 +246,14 @@ define([
                                     Source: "WsWorkunits.WUAction",
                                     Exceptions: [{Source: item.Action + " " + item.Wuid, Message: item.Result}]
                                 });
+                            } else {
+                                var wu = wuMap[item.Wuid];
+                                if (actionType === "delete" && item.Result === "Success") {
+                                    wu.set("StateID", 999);
+                                    wu.set("State", "deleted");
+                                } else if (wu.refresh) {
+                                    wu.refresh();
+                                }
                             }
                         });
                     }

initfiles/sbin/install-cluster.sh.in

@@ -62,7 +62,7 @@ getUserAndPasswd(){
        echo ""
 
        password_string="and a password ($(echo $PASS | sed 's/./\./g'))."
-       [ -z "$password" ] && password_string="with an empty password for passwordless login."
+       [ -z "$password_string" ] && password_string="with an empty password for passwordless login."
        echo "You entered user $ADMIN_USER $password_string"
        read -p  "Are these correct? [Y|n] " answer
        if [ "$answer" = "Y" ] || [ "$answer" = "y" ]

plugins/fileservices/fileservices.cpp

@@ -270,7 +270,7 @@ StringBuffer & constructLogicalName(ICodeContext * ctx, const char * partialLogi
     return constructLogicalName(wu, partialLogicalName, result);
 }
 
-static void WUmessage(ICodeContext *ctx, WUExceptionSeverity sev, const char *fn, const char *msg)
+static void WUmessage(ICodeContext *ctx, ErrorSeverity sev, const char *fn, const char *msg)
 {
     StringBuffer s("fileservices");
     if (fn)
@@ -331,7 +331,7 @@ FILESERVICES_API void FILESERVICES_CALL fsDeleteLogicalFile(ICodeContext *ctx, c
             s.append("') added to transaction");
         else
             s.append("') done");
-        WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+        WUmessage(ctx,SeverityInformation,NULL,s.str());
         AuditMessage(ctx,"DeleteLogicalFile",lfn.str());
 
     }
@@ -445,14 +445,14 @@ FILESERVICES_API void FILESERVICES_CALL fsRenameLogicalFile(ICodeContext *ctx, c
         queryDistributedFileDirectory().renamePhysical(lfn.str(),nlfn.str(),udesc,transaction);
         StringBuffer s("RenameLogicalFile ('");
         s.append(lfn).append(", '").append(nlfn).append("') done");
-        WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+        WUmessage(ctx,SeverityInformation,NULL,s.str());
         AuditMessage(ctx,"RenameLogicalFile",lfn.str(),nlfn.str());
     }
     catch (IException *e)
     {
         StringBuffer s;
         e->errorMessage(s);
-        WUmessage(ctx,ExceptionSeverityWarning,"RenameLogicalFile",s.str());
+        WUmessage(ctx,SeverityWarning,"RenameLogicalFile",s.str());
         throw e;
      }
 }
@@ -463,7 +463,7 @@ FILESERVICES_API void FILESERVICES_CALL fsSendEmail(ICodeContext * ctx, const ch
     StringArray warnings;
     sendEmail( to, subject, body, mailServer, port, sender, &warnings);
     ForEachItemIn(i,warnings)
-        WUmessage(ctx, ExceptionSeverityWarning, "SendEmail", warnings.item(i));
+        WUmessage(ctx, SeverityWarning, "SendEmail", warnings.item(i));
 }
 
 FILESERVICES_API void FILESERVICES_CALL fsSendEmailAttachText(ICodeContext * ctx, const char * to, const char * subject, const char * body, const char * attachment, const char * mimeType, const char * attachmentName, const char * mailServer, unsigned int port, const char * sender)
@@ -471,7 +471,7 @@ FILESERVICES_API void FILESERVICES_CALL fsSendEmailAttachText(ICodeContext * ctx
     StringArray warnings;
     sendEmailAttachText(to, subject, body, attachment, mimeType, attachmentName, mailServer, port, sender, &warnings);
     ForEachItemIn(i,warnings)
-        WUmessage(ctx, ExceptionSeverityWarning, "SendEmailAttachText", warnings.item(i));
+        WUmessage(ctx, SeverityWarning, "SendEmailAttachText", warnings.item(i));
 }
 
 FILESERVICES_API void FILESERVICES_CALL fsSendEmailAttachData(ICodeContext * ctx, const char * to, const char * subject, const char * body, size32_t lenAttachment, const void * attachment, const char * mimeType, const char * attachmentName, const char * mailServer, unsigned int port, const char * sender)
@@ -479,7 +479,7 @@ FILESERVICES_API void FILESERVICES_CALL fsSendEmailAttachData(ICodeContext * ctx
     StringArray warnings;
     sendEmailAttachData(to, subject, body, lenAttachment, attachment, mimeType, attachmentName, mailServer, port, sender, &warnings);
     ForEachItemIn(i,warnings)
-        WUmessage(ctx, ExceptionSeverityWarning, "SendEmailAttachData", warnings.item(i));
+        WUmessage(ctx, SeverityWarning, "SendEmailAttachData", warnings.item(i));
 }
 
 
@@ -585,7 +585,7 @@ static void blockUntilComplete(const char * label, IClientFileSpray &server, ICo
             {   //  Add warning of DFU Abort Request - should this be information  ---
                 StringBuffer s("DFU Workunit Abort Requested for ");
                 s.append(wuid);
-                WUmessage(ctx,ExceptionSeverityWarning,"blockUntilComplete",s.str());
+                WUmessage(ctx,SeverityWarning,"blockUntilComplete",s.str());
             }
 
             wu->setState(WUStateAborting);
@@ -1068,7 +1068,7 @@ static void CheckNotInTransaction(ICodeContext *ctx, const char *fn)
     if (transaction->active()) {
         StringBuffer s("Operation not part of transaction : ");
         s.append(fn);
-        WUmessage(ctx,ExceptionSeverityWarning,fn,s.str());
+        WUmessage(ctx,SeverityWarning,fn,s.str());
     }
 }
 
@@ -1083,7 +1083,7 @@ FILESERVICES_API void FILESERVICES_CALL fsCreateSuperFile(ICodeContext *ctx, con
     StringBuffer s("CreateSuperFile ('");
     s.append(lsfn).append("') done");
     AuditMessage(ctx,"CreateSuperFile",lsfn.str());
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
 }
 
 FILESERVICES_API bool FILESERVICES_CALL fsSuperFileExists(ICodeContext *ctx, const char *lsuperfn)
@@ -1112,7 +1112,7 @@ FILESERVICES_API void FILESERVICES_CALL fsDeleteSuperFile(ICodeContext *ctx, con
     } else {
         s.append(" file not found");
     }
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     if (found)
         AuditMessage(ctx,"DeleteSuperFile",lsfn.str());
 }
@@ -1181,7 +1181,7 @@ FILESERVICES_API void FILESERVICES_CALL fslStartSuperFileTransaction(ICodeContex
     IDistributedFileTransaction *transaction = ctx->querySuperFileTransaction();
     assertex(transaction);
     transaction->start();
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,"StartSuperFileTransaction");
+    WUmessage(ctx,SeverityInformation,NULL,"StartSuperFileTransaction");
 }
 
 FILESERVICES_API void FILESERVICES_CALL fsAddSuperFile(IGlobalCodeContext *gctx, const char *lsuperfn,const char *_lfn,unsigned atpos,bool addcontents, bool strict)
@@ -1257,7 +1257,7 @@ FILESERVICES_API void FILESERVICES_CALL fslAddSuperFile(ICodeContext *ctx, const
         s.append("trans");
     else
         s.append("done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"AddSuperFile",lsfn.str(),lfn.str());
 }
 
@@ -1298,7 +1298,7 @@ FILESERVICES_API void FILESERVICES_CALL fslRemoveSuperFile(ICodeContext *ctx, co
         s.append("trans");
     else
         s.append("done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"RemoveSuperFile",lsfn.str(),lfn.str());
 }
 
@@ -1337,7 +1337,7 @@ FILESERVICES_API void FILESERVICES_CALL fslRemoveOwnedSubFiles(ICodeContext *ctx
         s.append("trans");
     else
         s.append("done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"RemoveOwnedSubFiles",lsfn.str());
 }
 
@@ -1376,7 +1376,7 @@ FILESERVICES_API void FILESERVICES_CALL fslSwapSuperFile(ICodeContext *ctx, cons
         s.append("trans");
     else
         s.append("done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"SwapSuperFile",lsfn1.str(),lsfn2.str());
 }
 
@@ -1413,7 +1413,7 @@ FILESERVICES_API void FILESERVICES_CALL fslFinishSuperFileTransaction(ICodeConte
             s.append("rollback");
         else
             s.append("commit");
-        WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+        WUmessage(ctx,SeverityInformation,NULL,s.str());
     }
     else {
         StringBuffer s("Invalid FinishSuperFileTransaction ");
@@ -1422,7 +1422,7 @@ FILESERVICES_API void FILESERVICES_CALL fslFinishSuperFileTransaction(ICodeConte
         else
             s.append("done");
         s.append(", transaction not active");
-        WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+        WUmessage(ctx,SeverityInformation,NULL,s.str());
     }
 }
 
@@ -1471,12 +1471,12 @@ FILESERVICES_API char *  FILESERVICES_CALL fslWaitDfuWorkunit(ICodeContext *ctx,
     setServerAccess(server, wu);
     StringBuffer s("Waiting for DFU Workunit ");
     s.append(wuid);
-    WUmessage(ctx,ExceptionSeverityInformation,"WaitDfuWorkunit",s.str());
+    WUmessage(ctx,SeverityInformation,"WaitDfuWorkunit",s.str());
     StringBuffer state;
     wu.clear();
     blockUntilComplete("WaitDfuWorkunit", server, ctx, wuid, timeout, &state);
     s.clear().append("Finished waiting for DFU Workunit ").append(wuid).append(" state=").append(state.str());
-    WUmessage(ctx,ExceptionSeverityInformation,"WaitDfuWorkunit",s.str());
+    WUmessage(ctx,SeverityInformation,"WaitDfuWorkunit",s.str());
     return state.detach();
 }
 
@@ -1496,7 +1496,7 @@ FILESERVICES_API void FILESERVICES_CALL fslAbortDfuWorkunit(ICodeContext *ctx, c
     Linked<IClientAbortDFUWorkunitResponse> abortResp = server.AbortDFUWorkunit(abortReq);
     StringBuffer s("DFU Workunit Abort Requested for ");
     s.append(wuid);
-    WUmessage(ctx,ExceptionSeverityInformation,"AbortDfuWorkunit",s.str());
+    WUmessage(ctx,SeverityInformation,"AbortDfuWorkunit",s.str());
 }
 
 FILESERVICES_API void  FILESERVICES_CALL fsMonitorLogicalFileName(ICodeContext *ctx, const char *eventname, const char *_lfn,int shotcount, const char * espServerIpPort)
@@ -1522,7 +1522,7 @@ FILESERVICES_API char *  FILESERVICES_CALL fsfMonitorLogicalFileName(ICodeContex
     StringBuffer res(result->getWuid());
     StringBuffer s("MonitorLogicalFileName ('");
     s.append(lfn).append("'): ").append(res);
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     wu.clear();
     if (res.length()!=0)
         blockUntilComplete("MonitorLogicalFileName",server,ctx,res.str(),1000*60*60,NULL,true);
@@ -1552,7 +1552,7 @@ FILESERVICES_API char *  FILESERVICES_CALL fsfMonitorFile(ICodeContext *ctx, con
     StringBuffer res(result->getWuid());
     StringBuffer s("MonitorFile (");
     s.append(ip).append(", '").append(filename).append("'): '").append(res);
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     wu.clear();
     if (res.length()!=0)
         blockUntilComplete("MonitorFile",server,ctx,res.str(),1000*60*60,NULL,true);
@@ -1972,7 +1972,7 @@ FILESERVICES_API void FILESERVICES_CALL fsAddFileRelationship(ICodeContext * ctx
     queryDistributedFileDirectory().addFileRelationship(pfn.str(),sfn.str(),primflds,secflds,kind,cardinality,payload,ctx->queryUserDescriptor(), description);
     StringBuffer s("AddFileRelationship('");
     s.append(pfn.str()).append("','").append(sfn.str()).append("','").append(primflds?primflds:"").append("','").append(secflds?secflds:"").append("','").append(kind?kind:"").append("') done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
 
 }
 
@@ -2145,7 +2145,7 @@ FILESERVICES_API void  FILESERVICES_CALL fsMoveExternalFile(ICodeContext * ctx,c
     file->move(topath);
     StringBuffer s("MoveExternalFile ('");
     s.append(location).append(',').append(frompath).append(',').append(topath).append(") done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"MoveExternalFile",frompath,topath);
 }
 
@@ -2163,7 +2163,7 @@ FILESERVICES_API void  FILESERVICES_CALL fsDeleteExternalFile(ICodeContext * ctx
     file->remove();
     StringBuffer s("DeleteExternalFile ('");
     s.append(location).append(',').append(path).append(") done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"DeleteExternalFile",path);
 }
 
@@ -2181,7 +2181,7 @@ FILESERVICES_API void  FILESERVICES_CALL fsCreateExternalDirectory(ICodeContext
     file->createDirectory();
     StringBuffer s("CreateExternalDirectory ('");
     s.append(location).append(',').append(path).append(") done");
-    WUmessage(ctx,ExceptionSeverityInformation,NULL,s.str());
+    WUmessage(ctx,SeverityInformation,NULL,s.str());
     AuditMessage(ctx,"CreateExternalDirectory",path);
 }
 
@@ -2318,7 +2318,7 @@ FILESERVICES_API void FILESERVICES_CALL fsDfuPlusExec(ICodeContext * ctx,const c
         void info(const char *msg)
         {
             if (ctx&&(++limit<100))
-                WUmessage(ctx,ExceptionSeverityInformation,NULL,msg);
+                WUmessage(ctx,SeverityInformation,NULL,msg);
         }
         void err(const char *msg)
         {

roxie/ccd/ccd.hpp

@@ -44,9 +44,6 @@
 
 #define PARALLEL_EXECUTE
 
-#define RE_FailedToLoadProcedure    0x1000
-#define RE_FailedToLoadSharedObject 0x2000
-
 #define MAXTRACELEVEL 100     // don't want traceLevel+1 to wrap to 0 in lsb
 #define MAX_CLUSTER_SIZE 1024
 #define UDP_QUEUE_SIZE 100

roxie/ccd/ccdcontext.cpp

@@ -3564,9 +3564,9 @@ public:
 
     virtual void addWuException(const char * text, unsigned code, unsigned _severity, const char * source)
     {
-        WUExceptionSeverity severity = (WUExceptionSeverity) _severity;
+        ErrorSeverity severity = (ErrorSeverity) _severity;
         CTXLOG("%s", text);
-        if (severity > ExceptionSeverityInformation)
+        if (severity > SeverityInformation)
             OERRLOG("%d - %s", code, text);
         if (workUnit)
         {
@@ -3581,7 +3581,7 @@ public:
         if (workUnit)
         {
             WorkunitUpdate wu(&workUnit->lock());
-            addExceptionToWorkunit(wu, ExceptionSeverityError, "user", code, text, filename, lineno, column);
+            addExceptionToWorkunit(wu, SeverityError, "user", code, text, filename, lineno, column);
         }
         if (isAbort)
             rtlFailOnAssert();      // minimal implementation

roxie/ccd/ccddali.cpp

@@ -887,5 +887,5 @@ extern void addWuException(IConstWorkUnit *workUnit, IException *E)
     unsigned code = E->errorCode();
     OERRLOG("%u - %s", code, message.str());
     WorkunitUpdate w(&workUnit->lock());
-    addExceptionToWorkunit(w, ExceptionSeverityError, "Roxie", code, message.str(), NULL, 0, 0);
+    addExceptionToWorkunit(w, SeverityError, "Roxie", code, message.str(), NULL, 0, 0);
 }

roxie/ccd/ccdquery.cpp

@@ -296,6 +296,7 @@ QueryOptions::QueryOptions()
     skipFileFormatCrcCheck = false;
     stripWhitespaceFromStoredDataset = ((ptr_ignoreWhiteSpace & defaultXmlReadFlags) != 0);
     timeActivities = defaultTimeActivities;
+    allSortsMaySpill = false; // No global default for this
 }
 
 QueryOptions::QueryOptions(const QueryOptions &other)
@@ -319,7 +320,8 @@ QueryOptions::QueryOptions(const QueryOptions &other)
     enableFieldTranslation = other.enableFieldTranslation;
     skipFileFormatCrcCheck = other.skipFileFormatCrcCheck;
     stripWhitespaceFromStoredDataset = other.stripWhitespaceFromStoredDataset;
-    timeActivities =other.timeActivities;
+    timeActivities = other.timeActivities;
+    allSortsMaySpill = other.allSortsMaySpill;
 }
 
 void QueryOptions::setFromWorkUnit(IConstWorkUnit &wu, const IPropertyTree *stateInfo)
@@ -354,6 +356,7 @@ void QueryOptions::setFromWorkUnit(IConstWorkUnit &wu, const IPropertyTree *stat
     updateFromWorkUnit(skipFileFormatCrcCheck, wu, "skipFileFormatCrcCheck");
     updateFromWorkUnit(stripWhitespaceFromStoredDataset, wu, "stripWhitespaceFromStoredDataset");
     updateFromWorkUnit(timeActivities, wu, "timeActivities");
+    updateFromWorkUnit(allSortsMaySpill, wu, "allSortsMaySpill");
 }
 
 void QueryOptions::updateFromWorkUnitM(memsize_t &value, IConstWorkUnit &wu, const char *name)
@@ -398,6 +401,7 @@ void QueryOptions::setFromContext(const IPropertyTree *ctx)
         updateFromContext(skipFileFormatCrcCheck, ctx, "_SkipFileFormatCrcCheck", "@skipFileFormatCrcCheck");
         updateFromContext(stripWhitespaceFromStoredDataset, ctx, "_StripWhitespaceFromStoredDataset", "@stripWhitespaceFromStoredDataset");
         updateFromContext(timeActivities, ctx, "@timeActivities", "_TimeActivities");
+        // Note: allSortsMaySpill is not permitted at context level (too late anyway, unless I refactored)
     }
 }
 
@@ -719,7 +723,7 @@ protected:
         case TAKsoap_datasetaction:
             return createRoxieServerSoapDatasetActionActivityFactory(id, subgraphId, *this, helperFactory, kind, isRootAction(node));
         case TAKsort:
-            return createRoxieServerSortActivityFactory(id, subgraphId, *this, helperFactory, kind);
+            return createRoxieServerSortActivityFactory(id, subgraphId, *this, helperFactory, kind, node);
         case TAKspill:
         case TAKmemoryspillsplit:
             return createRoxieServerThroughSpillActivityFactory(id, subgraphId, *this, helperFactory, kind);
@@ -1043,6 +1047,7 @@ public:
         isLoadFailed = false;
         libraryInterfaceHash = 0;
         options.enableFieldTranslation = package.getEnableFieldTranslation();  // NOTE - can be overridden by wu settings
+        options.allSortsMaySpill = dynamic;
     }
 
     ~CQueryFactory()

roxie/ccd/ccdquery.hpp

@@ -115,6 +115,7 @@ public:
     bool skipFileFormatCrcCheck;
     bool stripWhitespaceFromStoredDataset;
     bool timeActivities;
+    bool allSortsMaySpill;
 
 private:
     static const char *findProp(const IPropertyTree *ctx, const char *name1, const char *name2);

roxie/ccd/ccdserver.cpp

@@ -7399,6 +7399,7 @@ interface ISortAlgorithm : extends IInterface
 
 class CQuickSortAlgorithm : implements CInterfaceOf<ISortAlgorithm>
 {
+protected:
     unsigned curIndex;
     ConstPointerArray sorted;
     ICompare *compare;
@@ -7432,6 +7433,24 @@ public:
     }
 };
 
+class CStableQuickSortAlgorithm : public CQuickSortAlgorithm
+{
+public:
+    CStableQuickSortAlgorithm(ICompare *_compare) : CQuickSortAlgorithm(_compare)
+    {
+    }
+    virtual void prepare(IRoxieInput *input)
+    {
+        curIndex = 0;
+        if (input->nextGroup(sorted))
+        {
+            unsigned size = sorted.ordinality();
+            MemoryAttr indexbuff(size*sizeof(void **));
+            qsortvecstable(const_cast<void * *>(sorted.getArray()), size, *compare, (void ***)indexbuff.bufferBase());
+        }
+    }
+};
+
 class CSpillingQuickSortAlgorithm : implements CInterfaceOf<ISortAlgorithm>, implements roxiemem::IBufferedRowCallback
 {
     enum {
@@ -7448,10 +7467,12 @@ class CSpillingQuickSortAlgorithm : implements CInterfaceOf<ISortAlgorithm>, imp
     Owned<IRowStream> diskReader;
     IOutputMetaData *rowMeta;
     unsigned activityId;
+    bool stable;
 
 public:
-    CSpillingQuickSortAlgorithm(ICompare *_compare, IRoxieSlaveContext * _ctx, IOutputMetaData * _rowMeta, unsigned _activityId)
-        : rowsToSort(&_ctx->queryRowManager(), InitialSortElements, CommitStep, _activityId), ctx(_ctx), compare(_compare), rowMeta(_rowMeta), activityId(_activityId)
+    CSpillingQuickSortAlgorithm(ICompare *_compare, IRoxieSlaveContext * _ctx, IOutputMetaData * _rowMeta, unsigned _activityId, bool _stable)
+        : rowsToSort(&_ctx->queryRowManager(), InitialSortElements, CommitStep, _activityId),
+          ctx(_ctx), compare(_compare), rowMeta(_rowMeta), activityId(_activityId), stable(_stable)
     {
         ctx->queryRowManager().addRowBuffer(this);
     }
@@ -7508,7 +7529,13 @@ public:
             {
                 const void * * rows = rowsToSort.getBlock(numRows);
                 //MORE: Should this be parallel?  Should that be dependent on whether it is grouped?  Should be a hint.
-                qsortvec(const_cast<void * *>(rows), numRows, *compare);
+                if (stable)
+                {
+                    MemoryAttr indexbuff(numRows*sizeof(void **));
+                    qsortvecstable(const_cast<void * *>(rows), numRows, *compare, (void ***)indexbuff.bufferBase());
+                }
+                else
+                    qsortvec(const_cast<void * *>(rows), numRows, *compare);
             }
             sorted.transferFrom(rowsToSort);
         }
@@ -8003,7 +8030,7 @@ public:
     }
 };
 
-typedef enum {heapSort, insertionSort, quickSort, spillingQuickSort, unknownSort } RoxieSortAlgorithm;
+typedef enum {heapSort, insertionSort, quickSort, stableQuickSort, spillingQuickSort, stableSpillingQuickSort, unknownSort } RoxieSortAlgorithm;
 
 class CRoxieServerSortActivity : public CRoxieServerActivity
 {
@@ -8037,8 +8064,12 @@ public:
         case quickSort:
             sorter.setown(new CQuickSortAlgorithm(compare));
             break;
+        case stableQuickSort:
+            sorter.setown(new CStableQuickSortAlgorithm(compare));
+            break;
         case spillingQuickSort:
-            sorter.setown(new CSpillingQuickSortAlgorithm(compare, ctx, meta, activityId));
+        case stableSpillingQuickSort:
+            sorter.setown(new CSpillingQuickSortAlgorithm(compare, ctx, meta, activityId, sortAlgorithm==stableSpillingQuickSort));
             break;
         case unknownSort:
             sorter.clear(); // create it later....
@@ -8077,9 +8108,13 @@ public:
                 {
                     if (stricmp(useAlgorithm, "quicksort")==0)
                     {
-                        if (sortFlags & TAFstable)
-                            throw MakeStringException(ROXIE_UNKNOWN_ALGORITHM, "Invalid stable sort algorithm %s requested", useAlgorithm.get());
-                        sorter.setown(new CQuickSortAlgorithm(compare));
+                        switch (sortFlags & (TAFstable|TAFspill))
+                        {
+                        case 0: sortAlgorithm = quickSort; break;
+                        case TAFstable: sortAlgorithm = stableQuickSort; break;
+                        case TAFspill: sortAlgorithm = spillingQuickSort; break;
+                        case TAFstable|TAFspill: sortAlgorithm = stableSpillingQuickSort; break;
+                        }
                     }
                     else if (stricmp(useAlgorithm, "heapsort")==0)
                         sorter.setown(new CHeapSortAlgorithm(compare));
@@ -8088,7 +8123,9 @@ public:
                     else
                     {
                         WARNLOG(ROXIE_UNKNOWN_ALGORITHM, "Ignoring unsupported sort order algorithm '%s', using default", useAlgorithm.get());
-                        if (sortFlags & TAFunstable)
+                        if (sortFlags & TAFspill)
+                            sorter.setown(new CSpillingQuickSortAlgorithm(compare, ctx, meta, activityId, (sortFlags & TAFstable) != 0));
+                        else if (sortFlags & TAFunstable)
                             sorter.setown(new CQuickSortAlgorithm(compare));
                         else
                             sorter.setown(new CHeapSortAlgorithm(compare));
@@ -8116,20 +8153,31 @@ class CRoxieServerSortActivityFactory : public CRoxieServerActivityFactory
 {
     RoxieSortAlgorithm sortAlgorithm;
     unsigned sortFlags;
-
+    bool forceSpill;
 public:
-    CRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind)
+    CRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind, IPropertyTree &_graphNode)
         : CRoxieServerActivityFactory(_id, _subgraphId, _queryFactory, _helperFactory, _kind)
     {
-        sortAlgorithm = heapSort;
-        sortFlags = TAFstable;
+        forceSpill = _queryFactory.queryOptions().allSortsMaySpill || _graphNode.getPropBool("hint[@name='spill']/@value", false);;
+        if (forceSpill)
+        {
+            sortAlgorithm = stableSpillingQuickSort;
+            sortFlags = TAFstable|TAFspill;
+        }
+        else
+        {
+            sortAlgorithm = heapSort;
+            sortFlags = TAFstable;
+        }
         Owned<IHThorSortArg> sortHelper = (IHThorSortArg *) helperFactory();
         IHThorAlgorithm *sortMethod = static_cast<IHThorAlgorithm *>(sortHelper->selectInterface(TAIalgorithm_1));
         if (sortMethod)
         {
             sortFlags = sortMethod->getAlgorithmFlags();
+            if (forceSpill)
+                sortFlags |= TAFspill;
             if (sortFlags & TAFspill)
-                sortAlgorithm = spillingQuickSort;
+                sortAlgorithm = stableSpillingQuickSort;
             else if (sortFlags & TAFunstable)
                 sortAlgorithm = quickSort;
             if (!(sortFlags & TAFconstant))
@@ -8141,18 +8189,16 @@ public:
                 {
                     if (stricmp(useAlgorithm, "quicksort")==0)
                     {
-                        if (sortFlags & TAFstable)
-                            throw MakeStringException(ROXIE_UNKNOWN_ALGORITHM, "Invalid stable sort algorithm %s requested", useAlgorithm.get());
-                        sortAlgorithm = quickSort;
-                    }
-                    else if (stricmp(useAlgorithm, "spillingquicksort")==0)
-                    {
-                        if (sortFlags & TAFstable)
-                            throw MakeStringException(ROXIE_UNKNOWN_ALGORITHM, "Invalid stable sort algorithm %s requested", useAlgorithm.get());
-                        sortAlgorithm = spillingQuickSort;
+                        switch (sortFlags & (TAFstable|TAFspill))
+                        {
+                        case 0: sortAlgorithm = quickSort; break;
+                        case TAFstable: sortAlgorithm = stableQuickSort; break;
+                        case TAFspill: sortAlgorithm = spillingQuickSort; break;
+                        case TAFstable|TAFspill: sortAlgorithm = stableSpillingQuickSort; break;
+                        }
                     }
                     else if (stricmp(useAlgorithm, "heapsort")==0)
-                        sortAlgorithm = heapSort; // NOTE - we do allow UNSTABLE('heapsort') in order to facilitate runtime selection
+                        sortAlgorithm = heapSort; // NOTE - we do allow UNSTABLE('heapsort') in order to facilitate runtime selection. Also explicit selection of heapsort overrides request to spill
                     else if (stricmp(useAlgorithm, "insertionsort")==0)
                         sortAlgorithm = insertionSort;
                     else
@@ -8174,9 +8220,9 @@ public:
     }
 };
 
-IRoxieServerActivityFactory *createRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind)
+IRoxieServerActivityFactory *createRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind, IPropertyTree &_graphNode)
 {
-    return new CRoxieServerSortActivityFactory(_id, _subgraphId, _queryFactory, _helperFactory, _kind);
+    return new CRoxieServerSortActivityFactory(_id, _subgraphId, _queryFactory, _helperFactory, _kind, _graphNode);
 }
 
 //=====================================================================================================

roxie/ccd/ccdserver.hpp

@@ -343,7 +343,7 @@ extern IRoxieServerActivityFactory *createRoxieServerRollupActivityFactory(unsig
 extern IRoxieServerActivityFactory *createRoxieServerNormalizeActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
 extern IRoxieServerActivityFactory *createRoxieServerNormalizeChildActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
 extern IRoxieServerActivityFactory *createRoxieServerNormalizeLinkedChildActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
-extern IRoxieServerActivityFactory *createRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
+extern IRoxieServerActivityFactory *createRoxieServerSortActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind, IPropertyTree &_graphNode);
 extern IRoxieServerActivityFactory *createRoxieServerThroughSpillActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
 extern IRoxieServerActivityFactory *createRoxieServerSplitActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);
 extern IRoxieServerActivityFactory *createRoxieServerPipeReadActivityFactory(unsigned _id, unsigned _subgraphId, IQueryFactory &_queryFactory, HelperFactory *_helperFactory, ThorActivityKind _kind);

system/include/errorlist.h

@@ -21,6 +21,11 @@
 // Define start and end ranges for error codes that may be thrown, written to a log, or returned in any manner
 // create a range that will be big enough
 
+//Error ranges used throughout the system...
+
+#define ERR_EXE_START           1     // All .exes can safely use the same range.
+#define ERR_EXE_END             999   // Codes 1000+ should be used in .dlls.
+
 // actual error codes do not belong in this file
 
 #define ERRORID_UNKNOWN         999      // Right now, this value is used to filter out error messages which have 
@@ -28,16 +33,17 @@
                                                     // example, if an error code is not greater than this value, ECLWatch displays 
                                                     // the error as "internal system error".
 
+//Some legacy ecl warnings appear in the following range, most new ones are in the range below
 #define ECL_WARN_START          1000
 #define ECL_WARN_END            1099
 
 #define ROXIEMM_ERROR_START     1300
 
-#define ROXIE_ERROR_START      1400   // roxie is already using this start value
-#define ROXIE_ERROR_END        1799
+#define ROXIE_ERROR_START       1400   // roxie is already using this start value
+#define ROXIE_ERROR_END         1799
 
-#define ROXIE_MGR_START        1800   // used in esp service to communicate / gather info for roxie
-#define ROXIE_MGR_END          1999
+#define ROXIE_MGR_START         1800   // used in esp service to communicate / gather info for roxie
+#define ROXIE_MGR_END           1999
 
 #define HQL_ERROR_START         2000    // 2000..3999 in hql, 4000..4999 in hqlcpp
 #define HQL_ERROR_END           4999
@@ -45,37 +51,54 @@
 #define WORKUNIT_ERROR_START    5000
 #define WORKUNIT_ERROR_END      5099
 
-#define PACKAGE_ERROR_START    5200
-#define PACKAGE_ERROR_END      5299
+#define PACKAGE_ERROR_START     5200
+#define PACKAGE_ERROR_END       5299
 
-#define THORHELPER_ERROR_START    5300
-#define THORHELPER_ERROR_END      5399
+#define THORHELPER_ERROR_START  5300
+#define THORHELPER_ERROR_END    5399
+
+#define ECLAGENT_ERROR_START    5400
+#define ECLAGENT_ERROR_END      5499
 
 #define WUWEB_ERROR_START       5500
 #define WUWEB_ERROR_END         5599
 
-#define XSLT_ERROR_START       5600
-#define XSLT_ERROR_END         5699
+#define XSLT_ERROR_START        5600
+#define XSLT_ERROR_END          5699
+
+#define THOR_ERROR_START        5700
+#define THOR_ERROR_END          5799
+
+//Jlib
+#define JLIB_ERROR_START        6000
+#define JLIB_ERROR_END          6499
+
+//Ecl runtime library
+#define ECLRTL_ERROR_START      6500
+#define ECLRTL_ERROR_END        6699
+
+//File view
+#define FILEVIEW_ERROR_START    6700
+#define FILEVIEW_ERROR_END      6749
 
 #define REMOTE_ERROR_START      8000    // dafilesrv etc - see common/remote/remoteerr.hpp
 #define REMOTE_ERROR_END        8099
 
-#define DISPATCH_ERROR_START   9000
-#define DISPATCH_ERROR_END     9399
+#define DISPATCH_ERROR_START    9000
+#define DISPATCH_ERROR_END      9399
 
 #define QUERYREGISTRY_ERROR_START   9600
 #define QUERYREGISTRY_ERROR_END     9799
 
-#define JVM_API_ERROR_START    10000
-#define JVM_API_ERROR_END      10499
+#define JVM_API_ERROR_START     10000
+#define JVM_API_ERROR_END       10499
 
-#define PKG_PROCESS_ERROR_START  11000
-#define PKG_PROCESS_ERROR_END    11100
+#define PKG_PROCESS_ERROR_START 11000
+#define PKG_PROCESS_ERROR_END   11100
 
 #define ECLWATCH_ERROR_START    20000
 #define ECLWATCH_ERROR_END      29999
 
 
-
 #endif
 

system/jlib/CMakeLists.txt

@@ -96,7 +96,6 @@ set (    INCLUDES
         jelogtype.hpp
         jencrypt.hpp
         jerror.hpp
-        jerrorrange.hpp
         jexcept.hpp
         jfile.hpp
         jfile.ipp

system/jlib/jcomp.cpp

@@ -23,6 +23,7 @@
 #include "jsem.hpp"
 #include "jexcept.hpp"
 #include "jregexp.hpp"
+#include "jerror.hpp"
 
 #ifdef _WIN32
 #include <windows.h>
@@ -564,9 +565,9 @@ void CppCompiler::extractErrors(IArrayOf<IError> & errors)
                 gccErrorPattern.findstr(msg, 5);
 
                 if (strieq(kind, "error"))
-                    errors.append(*createError(CategoryError, SeverityError, 2999, msg.str(), filename.str(), atoi(line), atoi(column), 0));
+                    errors.append(*createError(CategoryError, SeverityError, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), atoi(column), 0));
                 else
-                    errors.append(*createError(CategoryCpp, SeverityWarning, 2999, msg.str(), filename.str(), atoi(line), atoi(column), 0));
+                    errors.append(*createError(CategoryCpp, SeverityWarning, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), atoi(column), 0));
             }
             else if (gccErrorPattern2.find(next))
             {
@@ -577,9 +578,9 @@ void CppCompiler::extractErrors(IArrayOf<IError> & errors)
                 gccErrorPattern2.findstr(msg, 4);
 
                 if (strieq(kind, "error"))
-                    errors.append(*createError(CategoryError, SeverityError, 2999, msg.str(), filename.str(), atoi(line), 0, 0));
+                    errors.append(*createError(CategoryError, SeverityError, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), 0, 0));
                 else
-                    errors.append(*createError(CategoryCpp, SeverityWarning, 2999, msg.str(), filename.str(), atoi(line), 0, 0));
+                    errors.append(*createError(CategoryCpp, SeverityWarning, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), 0, 0));
             }
             else if (gccLinkErrorPattern.find(next))
             {
@@ -589,14 +590,14 @@ void CppCompiler::extractErrors(IArrayOf<IError> & errors)
                 gccLinkErrorPattern.findstr(msg, 3);
 
                 ErrorSeverity severity = linkFailed ? SeverityError : SeverityWarning;
-                errors.append(*createError(CategoryError, severity, 2999, msg.str(), filename.str(), atoi(line), 0, 0));
+                errors.append(*createError(CategoryError, severity, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), 0, 0));
             }
             else if (gccLinkErrorPattern2.find(next))
             {
                 StringBuffer msg("C++ link error: ");
                 gccLinkErrorPattern2.findstr(msg, 1);
                 ErrorSeverity severity = linkFailed ? SeverityError : SeverityWarning;
-                errors.append(*createError(CategoryError, severity, 2999, msg.str(), NULL, 0, 0, 0));
+                errors.append(*createError(CategoryError, severity, JLIBERR_CppCompileError, msg.str(), NULL, 0, 0, 0));
             }
             else if (vsErrorPattern.find(next))
             {
@@ -604,14 +605,14 @@ void CppCompiler::extractErrors(IArrayOf<IError> & errors)
                 vsErrorPattern.findstr(filename, 1);
                 vsErrorPattern.findstr(line, 2);
                 vsErrorPattern.findstr(msg, 3);
-                errors.append(*createError(CategoryError, SeverityError, 2999, msg.str(), filename.str(), atoi(line), 0, 0));
+                errors.append(*createError(CategoryError, SeverityError, JLIBERR_CppCompileError, msg.str(), filename.str(), atoi(line), 0, 0));
             }
             else if (vsLinkErrorPattern.find(next))
             {
                 StringBuffer filename, msg("C++ link error: ");
                 vsLinkErrorPattern.findstr(filename, 1);
                 vsLinkErrorPattern.findstr(msg, 2);
-                errors.append(*createError(CategoryError, SeverityError, 2999, msg.str(), filename.str(), 0, 0, 0));
+                errors.append(*createError(CategoryError, SeverityError, JLIBERR_CppCompileError, msg.str(), filename.str(), 0, 0, 0));
             }
         } while (cur);
     }

system/jlib/jerror.hpp

@@ -20,13 +20,18 @@
 #define JERROR_HPP
 
 #include "jexcept.hpp"
-#include "jerrorrange.hpp"
+#include "errorlist.h"
+
+#if (JLIB_ERROR_START != 6000 || JLIB_ERROR_END != 6499)
+#error "JLIB_ERROR_START has changed"
+#endif
 
 /* Errors generated in jlib */
 
-#define JLIBERR_BadlyFormedDateTime             1000
-#define JLIBERR_BadUtf8InArguments              1001
-#define JLIBERR_InternalError                   1002
+#define JLIBERR_BadlyFormedDateTime             6000
+#define JLIBERR_BadUtf8InArguments              6001
+#define JLIBERR_InternalError                   6002
+#define JLIBERR_CppCompileError                 6003
 
 //---- Text for all errors (make it easy to internationalise) ---------------------------
 

system/jlib/jexcept.hpp

@@ -162,17 +162,20 @@ void  jlib_decl printStackReport();
 
 //---------------------------------------------------------------------------------------------------------------------
 
+//These values are persisted - so should not be reordered.
 enum ErrorSeverity
 {
-    SeverityIgnore,
-    SeverityInfo,
-    SeverityWarning,
-    SeverityError,    // a warning treated as an error
-    SeverityFatal,      // a fatal error - can't be mapped to anything else
+    SeverityInformation = 0,
+    SeverityWarning = 1,
+    SeverityError = 2,
+    SeverityAlert = 3,
+    SeverityIgnore = 4,
+    SeverityFatal = 5,      // a fatal error - can't be mapped to anything else
     SeverityUnknown,
+    SeverityMax = SeverityUnknown
 };
 
-inline bool isError(ErrorSeverity severity) { return severity >= SeverityError; }
+inline bool isError(ErrorSeverity severity) { return severity == SeverityError || severity == SeverityFatal; }
 inline bool isFatal(ErrorSeverity severity) { return severity == SeverityFatal; }
 
 //TBD in a separate commit - add support for warnings to be associated with different categories

system/jlib/jliball.hpp

@@ -20,6 +20,7 @@
 #define JLIBALL_HPP
 
 #include "platform.h"
+#include "errorlist.h"
 #include "jiface.hpp"
 
 #include "jarray.hpp"
@@ -29,7 +30,6 @@
 #include "jcrc.hpp"
 #include "jdebug.hpp"
 #include "jencrypt.hpp"
-#include "jerrorrange.hpp"
 #include "jexcept.hpp"
 #include "jfile.hpp"
 #include "jhash.hpp"

testing/regress/README.rst

@@ -583,7 +583,8 @@ The format of the output is the same as 'run', except there is a log, result and
     To allow multiple tests to be generated from a single source file
     The regression suite engine executes the file once for each //version line in the file. It is compiled with command line option -Dn1=v1 -Dn2=v2 etc.
     The string value should quoted with \'.
-//version <n1>=<v1>,<n2>=<v2>,...
+    Optionally 'no<target>' exclusion info can add at the end of tag.
+//version <n1>=<v1>,<n2>=<v2>,...[,no<target>[,no<target>]]
 
 7. Key file handling:
 ---------------------

testing/regress/ecl/HeadingExample.ecl

@@ -0,0 +1,12 @@
+
+//Sample data
+rec := record
+	string s1;
+	string s2;
+end;
+ds0 := dataset([{'xxx','yyy'},{'xx,x','y,yy'},{'zzz','z'}],rec);
+//Output dataset
+ds0;
+
+//Save dataset to file with heading option
+output(ds0,,'~thor::jas::testout',csv(heading,SEPARATOR(','),quote('"')),overwrite,expire);

testing/regress/ecl/all_denormalize.ecl

@@ -26,6 +26,8 @@ useLocal := #IFDEFINED(root.useLocal, false);
 
 //--- end of version configuration ---
 
+#onwarning (4523, ignore);
+
 import $.setup;
 files := setup.files(multiPart, useLocal);
 

testing/regress/ecl/bug12130.ecl

@@ -30,6 +30,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4522, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/bug5236.ecl

@@ -30,6 +30,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4522, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/canmatch.ecl

@@ -30,6 +30,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/childindex.ecl

@@ -31,6 +31,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 
 #option ('layoutTranslationEnabled', useTranslation);
 #onwarning (4515, ignore);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/countindex.ecl

@@ -31,6 +31,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 
 #option ('layoutTranslationEnabled', useTranslation);
 #onwarning (4515, ignore);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/dict_keyed.ecl

@@ -26,6 +26,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 
 //--- end of version configuration ---
 
+#onwarning (4523, ignore);
+
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);
 

testing/regress/ecl/fetch.ecl

@@ -30,6 +30,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/fullkeyed.ecl

@@ -28,6 +28,7 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4522, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/hthor/layouttrans_disabled.xml

@@ -0,0 +1 @@
+<Exception><Source>eclagent</Source><Message>System error: 0: Index layout does not match published layout for index regress::multi::dg_fetchindex1</Message></Exception>

testing/regress/ecl/indexagg.ecl

@@ -19,6 +19,7 @@
 //class=index
 //version multiPart=false
 //version multiPart=true
+//version multiPart=true,useLocal=true
 //version multiPart=true,useTranslation=true,nothor
 
 import ^ as root;
@@ -29,14 +30,14 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);
 
 sequential(
 
-output(max(choosen(Files.DG_FetchIndex, 1,3), Fname));
-output(min(choosen(Files.DG_FetchIndex, 1,3), Fname));
-
-output(count(choosen(Files.DG_FetchIndex, 1,3)))
+output(max(Files.DG_FetchIndex, Fname));
+output(min(Files.DG_FetchIndex, Fname));
+output(count(Files.DG_FetchIndex))
 );

system/jlib/jerrorrange.hpp

@@ -15,17 +15,28 @@
     limitations under the License.
 ############################################################################## */
 
+//class=file
+//class=index
+//version multiPart=false
+//version multiPart=true
 
-#ifndef JERRORRANGE_HPP
-#define JERRORRANGE_HPP
+import ^ as root;
+multiPart := #IFDEFINED(root.multiPart, true);
+useLocal := #IFDEFINED(root.useLocal, false);
+useTranslation := #IFDEFINED(root.useTranslation, false);
 
-//Error ranges used throughout the system...
+//--- end of version configuration ---
 
-#define ERR_EXE_FIRST       1     // All .exes can safely use the same range.
-#define ERR_EXE_LAST        999   // Codes 1000+ should be used in .dlls.
+#option ('layoutTranslationEnabled', useTranslation);
+#onwarning (5402, ignore);
 
-#define ERR_JLIB_FIRST      1000
-#define ERR_JLIB_LAST       1999
-#define ERR_FILEVIEW_FIRST  2000
-#define ERR_FILEVIEW_LAST   2049
-#endif
+import $.setup;
+Files := setup.Files(multiPart, useLocal, useTranslation);
+
+sequential(
+
+output(max(choosen(Files.DG_FetchIndex, 1,3), Fname));
+output(min(choosen(Files.DG_FetchIndex, 1,3), Fname));
+
+output(count(choosen(Files.DG_FetchIndex, 1,3)))
+);

testing/regress/ecl/indexread.ecl

@@ -29,6 +29,7 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);
@@ -39,25 +40,6 @@ IMPORT Std;
 
 o1 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),1,SKIP), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
 o2 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),10,SKIP), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
-o3 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),1,SKIP,KEYED), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
-o4 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),10,SKIP,KEYED), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
 
-iresult := Files.DG_FetchIndex(Lname IN ['Anderson', 'Taylor']);
-
-lkresult := LIMIT(iresult,10,KEYED);
-lsresult := LIMIT(lkresult,10,SKIP);
-sresult := IF(Std.System.Thorlib.platform() != 'hthor', SORT(lsresult,Lname), lsresult);
-o5 := output(sresult, {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
-
-// then try with a keyed and unkeyed....
-
-o6 := output(LIMIT(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),1,SKIP,keyed),1,skip), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
-o7 := output(LIMIT(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),10,SKIP,keyed),10,skip), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
-
- o1:independent;
- o2:independent;
- o3:independent;
- o4:independent;
- o5:independent;
- o6:independent;
- o7:independent;
+o1:independent;
+o2:independent;

testing/regress/ecl/indexread2.ecl

@@ -30,6 +30,9 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4522, ignore);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/indexread3.ecl

@@ -30,8 +30,10 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4523, ignore);
 #onwarning (4527, ignore);
 #onwarning (4528, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/indexread4.ecl

@@ -30,6 +30,7 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/indexread6.ecl

@@ -30,6 +30,8 @@ useTranslation := #IFDEFINED(root.useTranslation, false);
 //--- end of version configuration ---
 
 #option ('layoutTranslationEnabled', useTranslation);
+#onwarning (4523, ignore);
+#onwarning (5402, ignore);
 
 import $.setup;
 Files := setup.Files(multiPart, useLocal, useTranslation);

testing/regress/ecl/indexread_keyed.ecl

@@ -0,0 +1,58 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2012 HPCC Systems.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+############################################################################## */
+
+//class=index
+//version multiPart=false
+//version multiPart=true
+//version multiPart=true,useLocal=true
+
+import ^ as root;
+multiPart := #IFDEFINED(root.multiPart, true);
+useLocal := #IFDEFINED(root.useLocal, false);
+useTranslation := false;    // keyed limits do not produce the same results.
+
+//--- end of version configuration ---
+
+#onwarning (5402, ignore);
+
+import $.setup;
+Files := setup.Files(multiPart, useLocal, useTranslation);
+
+IMPORT Std;
+
+// try it with just one limit
+
+o3 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),1,SKIP,KEYED), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
+o4 := output(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),10,SKIP,KEYED), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
+
+iresult := Files.DG_FetchIndex(Lname IN ['Anderson', 'Taylor']);
+
+lkresult := LIMIT(iresult,10,KEYED);
+lsresult := LIMIT(lkresult,10,SKIP);
+sresult := IF(useTranslation OR Std.System.Thorlib.platform() != 'hthor', SORT(lsresult,Lname), lsresult);
+o5 := output(sresult, {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
+
+// then try with a keyed and unkeyed....
+
+o6 := output(LIMIT(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),1,SKIP,keyed),1,skip), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
+o7 := output(LIMIT(LIMIT(Files.DG_FetchIndex(Lname='Anderson'),10,SKIP,keyed),10,skip), {Lname,Fname,TRIM(tfn),state,TRIM(blobfield)});
+
+ o3:independent;
+ o4:independent;
+ o5:independent;
+ o6:independent;
+ o7:independent;

testing/regress/ecl/key/HeadingExample.xml

@@ -0,0 +1,7 @@
+<Dataset name='Result 1'>
+ <Row><s1>xxx</s1><s2>yyy</s2></Row>
+ <Row><s1>xx,x</s1><s2>y,yy</s2></Row>
+ <Row><s1>zzz</s1><s2>z</s2></Row>
+</Dataset>
+<Dataset name='Result 2'>
+</Dataset>

testing/regress/ecl/key/indexagg.xml

@@ -1,9 +1,9 @@
 <Dataset name='Result 1'>
- <Row><Result_1>John           </Result_1></Row>
+ <Row><Result_1>Zeek           </Result_1></Row>
 </Dataset>
 <Dataset name='Result 2'>
- <Row><Result_2>John           </Result_2></Row>
+ <Row><Result_2>Bridge         </Result_2></Row>
 </Dataset>
 <Dataset name='Result 3'>
- <Row><Result_3>1</Result_3></Row>
+ <Row><Result_3>22</Result_3></Row>
 </Dataset>

testing/regress/ecl/key/indexagg_choosen.xml

@@ -0,0 +1,9 @@
+<Dataset name='Result 1'>
+ <Row><Result_1>John           </Result_1></Row>
+</Dataset>
+<Dataset name='Result 2'>
+ <Row><Result_2>John           </Result_2></Row>
+</Dataset>
+<Dataset name='Result 3'>
+ <Row><Result_3>1</Result_3></Row>
+</Dataset>

testing/regress/ecl/key/indexread.xml

@@ -7,29 +7,3 @@
  <Row><lname>Anderson                 </lname><fname>Larry          </fname><_unnamed_3>Larry</_unnamed_3><state>FL</state><_unnamed_5>Larry          Anderson</_unnamed_5></Row>
  <Row><lname>Anderson                 </lname><fname>Sue            </fname><_unnamed_3>Sue</_unnamed_3><state>FL</state><_unnamed_5>Sue            Anderson</_unnamed_5></Row>
 </Dataset>
-<Dataset name='Result 3'>
-</Dataset>
-<Dataset name='Result 4'>
- <Row><lname>Anderson                 </lname><fname>Jane           </fname><_unnamed_3>Jane</_unnamed_3><state>FL</state><_unnamed_5>Jane           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Joe            </fname><_unnamed_3>Joe</_unnamed_3><state>FL</state><_unnamed_5>Joe            Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>John           </fname><_unnamed_3>John</_unnamed_3><state>FL</state><_unnamed_5>John           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Larry          </fname><_unnamed_3>Larry</_unnamed_3><state>FL</state><_unnamed_5>Larry          Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Sue            </fname><_unnamed_3>Sue</_unnamed_3><state>FL</state><_unnamed_5>Sue            Anderson</_unnamed_5></Row>
-</Dataset>
-<Dataset name='Result 5'>
- <Row><lname>Anderson                 </lname><fname>Jane           </fname><_unnamed_3>Jane</_unnamed_3><state>FL</state><_unnamed_5>Jane           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Joe            </fname><_unnamed_3>Joe</_unnamed_3><state>FL</state><_unnamed_5>Joe            Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>John           </fname><_unnamed_3>John</_unnamed_3><state>FL</state><_unnamed_5>John           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Larry          </fname><_unnamed_3>Larry</_unnamed_3><state>FL</state><_unnamed_5>Larry          Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Sue            </fname><_unnamed_3>Sue</_unnamed_3><state>FL</state><_unnamed_5>Sue            Anderson</_unnamed_5></Row>
- <Row><lname>Taylor                   </lname><fname>Frank          </fname><_unnamed_3>Frank</_unnamed_3><state>FL</state><_unnamed_5>Frank          Taylor</_unnamed_5></Row>
-</Dataset>
-<Dataset name='Result 6'>
-</Dataset>
-<Dataset name='Result 7'>
- <Row><lname>Anderson                 </lname><fname>Jane           </fname><_unnamed_3>Jane</_unnamed_3><state>FL</state><_unnamed_5>Jane           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Joe            </fname><_unnamed_3>Joe</_unnamed_3><state>FL</state><_unnamed_5>Joe            Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>John           </fname><_unnamed_3>John</_unnamed_3><state>FL</state><_unnamed_5>John           Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Larry          </fname><_unnamed_3>Larry</_unnamed_3><state>FL</state><_unnamed_5>Larry          Anderson</_unnamed_5></Row>
- <Row><lname>Anderson                 </lname><fname>Sue            </fname><_unnamed_3>Sue</_unnamed_3><state>FL</state><_unnamed_5>Sue            Anderson</_unnamed_5></Row>
-</Dataset>