Merge branch 'master' into candidate-8.0.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>

docs/EN_US/ContainerizedHPCC/ContainerizedHPCCSystemsPlatform.xml

@@ -69,7 +69,7 @@
     deployments. This is useful for cloud-based deployments (large or small)
     or local testing/development deployments.</para>
 
-    <para>Docker containers managed by Kubernetes (k8s) is a new target
+    <para>Docker containers managed by Kubernetes (K8s) is a new target
     operating environment, alongside continued support for traditional “bare
     metal” installations using .deb or .rpm installer files. Support for
     traditional installers continues and that type of deployment is viable for
@@ -134,8 +134,8 @@
         process and specifying on which physical machines they should
         run.</para>
 
-        <para>In a Kubernetes (k8s) world, this is managed dynamically by the
-        K8s system itself (and can be changed dynamically as the system
+        <para>In a Kubernetes world, this is managed dynamically by the K8s
+        system itself (and can be changed dynamically as the system
         runs).</para>
 
         <para>Additionally, a containerized system is much simpler to manage
@@ -151,7 +151,7 @@
         <para>In the containerized world, the information that the operator
         needs to supply to configure an HPCC Systems environment is greatly
         reduced. There is no need to specify any information about what
-        machines are in use by what process, as mentioned above, but there is
+        machines are in use by what process, as mentioned above, and there is
         also no need to change a lot of options that might be dependent on the
         operating environment, since much of that was standardized at the time
         the container images were built.</para>
@@ -183,18 +183,18 @@
 
         <para>For example, an eclccserver component launches a stub requiring
         minimal resources, where the sole task is to watch for workunits
-        submitted for compilation and launch an independent k8s job to perform
+        submitted for compilation and launch an independent K8s job to perform
         the actual compile.</para>
 
         <para>Similarly, the eclagent component is also a stub that launches a
-        k8s job when a workunit is submitted and the Thor stub starts up a
+        K8s job when a workunit is submitted and the Thor stub starts up a
         Thor cluster only when required. Using this design, not only does the
         capacity of the system automatically scale up to use as many pods as
         needed to handle the submitted load, it scales down to use minimal
         resources (as little as a fraction of a single node) during idle times
         when waiting for jobs to be submitted.</para>
 
-        <para>ESP and Dali components are always-on as long as the k8s cluster
+        <para>ESP and Dali components are always-on as long as the K8s cluster
         is started--it isn’t feasible to start and stop them on demand without
         excessive latency. However, ESP can be scaled up and down dynamically
         to run as many instances needed to handle the current load.</para>

docs/EN_US/ContainerizedHPCC/ContainerizedMods/LocalDeployment.xml

@@ -5,8 +5,7 @@
   <title>Local Deployment (Development and Testing)</title>
 
   <para>While there are many ways to install a local single node HPCC Systems
-  Platform, this section focuses on using Docker Desktop with WSL 2 or
-  Hyper-V.</para>
+  Platform, this section focuses on using Docker Desktop.</para>
 
   <sect1 id="prereq">
     <title>Prerequisites</title>
@@ -46,6 +45,22 @@
         <para>Install Helm</para>
       </listitem>
     </itemizedlist>
+
+    <para>OR</para>
+
+    <itemizedlist>
+      <listitem>
+        <para>Install Docker Desktop on macOS</para>
+      </listitem>
+
+      <listitem>
+        <para>Enable Kubernetes in Docker Desktop</para>
+      </listitem>
+
+      <listitem>
+        <para>Install Helm </para>
+      </listitem>
+    </itemizedlist>
   </sect1>
 
   <sect1 id="addrepo">
@@ -75,11 +90,26 @@
     <para>Expected response:</para>
 
     <para><programlisting>NAME: mycluster
-LAST DEPLOYED: Wed Jan 20 11:50:56 2021
+LAST DEPLOYED: Tue Mar 23 13:26:55 2021
 NAMESPACE: default
 STATUS: deployed
 REVISION: 1
-TEST SUITE: None</programlisting></para>
+TEST SUITE: None
+NOTES:
+Thank you for installing the HPCC chart.
+
+This chart has defined the following HPCC components:
+  dali.mydali
+  eclagent.hthor
+  eclagent.roxie-workunit
+  eclccserver.myeclccserver
+  esp.eclwatch
+  esp.eclservices
+  esp.eclqueries
+  esp.esdl-sandbox
+  esp.sql2ecl
+  roxie.roxie
+  thor.thor</programlisting></para>
 
     <para><emphasis role="bold">To check status:</emphasis></para>
 
@@ -87,14 +117,27 @@ TEST SUITE: None</programlisting></para>
 
     <para>Expected response:</para>
 
-    <para><programlisting>NAME                            READY STATUS   RESTARTS AGE
-hthor-56bd86f4b9-h5746          1/1   Running  0        30s
-mydali-7f669fd7d6-frjbh         1/1   Running  0        30s
-myeclccserver-8665655fc8-krh8r  1/1   Running  0        30s
-myesp-77f4c69ddf-chjrn          1/1   Running  0        30s
-roxie-bb8dcdcd9-6gmlj           1/1   Running  0        30s
-thor-agent-797fdf698f-lmlzw     1/1   Running  0        30s
-thor-thoragent-974c58fbb-c5cd4  1/1   Running  0        30s</programlisting></para>
+    <para><programlisting>NAME                                          READY   STATUS    RESTARTS   AGE
+eclqueries-7fd94d77cb-m7lmb                   1/1     Running   0          2m6s
+eclservices-b57f9b7cc-bhwtm                   1/1     Running   0          2m6s
+eclwatch-599fb7845-2hq54                      1/1     Running   0          2m6s
+esdl-sandbox-848b865d46-9bv9r                 1/1     Running   0          2m6s
+hthor-745f598795-ql9dl                        1/1     Running   0          2m6s
+mydali-6b844bfcfb-jv7f6                       2/2     Running   0          2m6s
+myeclccserver-75bcc4d4d-gflfs                 1/1     Running   0          2m6s
+roxie-agent-1-77f696466f-tl7bb                1/1     Running   0          2m6s
+roxie-agent-1-77f696466f-xzrtf                1/1     Running   0          2m6s
+roxie-agent-2-6dd45b7f9d-m22wl                1/1     Running   0          2m6s
+roxie-agent-2-6dd45b7f9d-xmlmk                1/1     Running   0          2m6s
+roxie-toposerver-695fb9c5c7-9lnp5             1/1     Running   0          2m6s
+roxie-workunit-d7446699f-rvf2z                1/1     Running   0          2m6s
+sasha-dfurecovery-archiver-78c47c4db7-k9mdz   1/1     Running   0          2m6s
+sasha-dfuwu-archiver-576b978cc7-b47v7         1/1     Running   0          2m6s
+sasha-file-expiry-8496d87879-xct7f            1/1     Running   0          2m6s
+sasha-wu-archiver-5f64594948-xjblh            1/1     Running   0          2m6s
+sql2ecl-5c8c94d55-tj4td                       1/1     Running   0          2m6s
+thor-eclagent-6b8f564f9c-qnczz                1/1     Running   0          2m6s
+thor-thoragent-56d788869f-7trxk               1/1     Running   0          2m6s</programlisting></para>
 
     <para><variablelist>
         <varlistentry>
@@ -134,16 +177,18 @@ thor-thoragent-974c58fbb-c5cd4  1/1   Running  0        30s</programlisting></pa
 
     <para>Expected response:</para>
 
-    <para><programlisting>NAME             TYPE         CLUSTER-IP      EXTERNAL-IP  PORT(S) AGE
-eclqueries       LoadBalancer 10.108.171.35   localhost    8002:31615/TCP 3m40s
-eclservices      ClusterIP    10.107.121.158  &lt;none&gt;       8010/TCP 3m40s
-eclwatch         LoadBalancer 10.100.81.69    localhost    8010:30173/TCP 3m40s
-esdl-sandbox     LoadBalancer 10.100.194.33   localhost    8899:30705/TCP 3m40s
-kubernetes       ClusterIP    10.96.0.1       &lt;none&gt;       443/TCP 20h
-mydali           ClusterIP    10.102.80.158   &lt;none&gt;       7070/TCP 3m40s
-roxie            LoadBalancer 10.100.134.125  localhost    9876:30480/TCP 3m40s
-roxie-toposerver ClusterIP    None            &lt;none&gt;       9004/TCP 3m40s
-sql2ecl          LoadBalancer 10.107.177.180  localhost    8510:30054/TCP 3m40s</programlisting></para>
+    <para><programlisting>NAME                  TYPE         CLUSTER-IP      EXTERNAL-IP  PORT(S)           AGE
+eclqueries            LoadBalancer 10.108.171.35   localhost    8002:31615/TCP    2m6s
+eclservices           ClusterIP    10.107.121.158  &lt;none&gt;       8010/TCP          2m6s
+eclwatch              LoadBalancer 10.100.81.69    localhost    8010:30173/TCP    2m6s
+esdl-sandbox          LoadBalancer 10.100.194.33   localhost    8899:30705/TCP    2m6s
+kubernetes            ClusterIP    10.96.0.1       &lt;none&gt;       443/TCP           2m6s
+mydali                ClusterIP    10.102.80.158   &lt;none&gt;       7070/TCP          2m6s
+roxie                 LoadBalancer 10.100.134.125  localhost    9876:30480/TCP    2m6s
+roxie-toposerver      ClusterIP    None            &lt;none&gt;       9004/TCP          2m6s
+sasha-dfuwu-archiver  ClusterIP    10.110.200.110  &lt;none&gt;       8877/TCP          2m6s
+sasha-wu-archiver     ClusterIP    10.111.34.240   &lt;none&gt;       8877/TCP          2m6s
+sql2ecl               LoadBalancer 10.107.177.180  localhost    8510:30054/TCP    2m6s</programlisting></para>
 
     <para>Notice the <emphasis role="strong">eclwatch</emphasis> service is
     running on <emphasis role="strong">localhost:8010</emphasis>. Use that

docs/EN_US/ContainerizedHPCC/ContainerizedMods/PVCStorage.xml

@@ -13,19 +13,24 @@
     typically removed when the cluster is stopped. This is fine for testing
     but for any real application, you want persistent storage.</para>
 
-    <para>To persist data on a Windows machine with Docker Desktop, the first
-    step is to make sure the relevant directories exist:</para>
-
-    <para></para>
+    <para>To persist data with Docker Desktop, the first step is to make sure
+    the relevant directories exist:</para>
 
     <orderedlist>
       <listitem>
-        <para>Create data directories using a Windows command prompt:</para>
+        <para>Create data directories using a terminal interface: </para>
+
+        <para>For Windows, use this command:</para>
 
         <para><programlisting>mkdir c:\hpccdata
 mkdir c:\hpccdata\dlls
 mkdir c:\hpccdata\data
-mkdir c:\hpccdata\dali</programlisting></para>
+mkdir c:\hpccdata\dali
+</programlisting></para>
+
+        <para>For macOS, use this command:</para>
+
+        <para><programlisting>mkdir -p /Users/myUser/hpccdata/{dlls,data,dali} </programlisting></para>
       </listitem>
 
       <listitem>
@@ -38,8 +43,6 @@ mkdir c:\hpccdata\dali</programlisting></para>
         <para>If you want only the helm charts, the use the <emphasis
         role="strong">helm-chart</emphasis> repository
         (https://github.com/hpcc-systems/helm-chart).</para>
-
-        <para></para>
       </listitem>
 
       <listitem>
@@ -59,7 +62,11 @@ helm install localfile examples/local/hpcc-localfile
            --set common.hostpath=/run/desktop/mnt/host/c/hpccdata
 
 # for a Hyper-V deployment:
-helm install localfile examples/local/hpcc-localfile --set common.hostpath=/c/hpccdata</programlisting></para>
+helm install localfile examples/local/hpcc-localfile --set common.hostpath=/c/hpccdata
+
+# for a macOS deployment:
+helm install localfile examples/local/hpcc-localfile --set common.hostpath=/Users/myUser/hpccdata
+</programlisting></para>
 
         <para>The <emphasis role="bold">--set</emphasis> option specifies the
         base directory:</para>
@@ -69,12 +76,26 @@ helm install localfile examples/local/hpcc-localfile --set common.hostpath=/c/hp
         access to the host file system for WSL2.</para>
 
         <para>The path <emphasis role="bold">/c/hpccdata</emphasis> provides
-        access to the host file system for Hyper-V</para>
+        access to the host file system for Hyper-V.</para>
+
+        <para>The path <emphasis role="bold">/Users/myUser/hpccdata</emphasis>
+        provides access to the host file system for Mac OSX.</para>
+      </listitem>
+
+      <listitem>
+        <para>If you are using Docker Desktop with Hyper-V, add the shared
+        data folder (in this example, C:\hpccdata) in the Docker Desktop
+        settings. </para>
+
+        <para>This is <emphasis role="bold">not</emphasis> needed in a macOS
+        or WSL 2 environment. </para>
+
+        <graphic fileref="../../images/dockerFileShare.jpg" />
       </listitem>
 
       <listitem>
-        <para>Finally, install the hpcc chart, and provide a yaml file that
-        provides storage information that uses the PVCs created by the
+        <para>Finally, install the hpcc Helm chart, and provide a yaml file
+        that provides storage information that uses the PVCs created by the
         previous step.</para>
 
         <para>The example directory contains a sample yaml file that can be

esp/src/src-react/components/Title.tsx

@@ -1,5 +1,5 @@
 import * as React from "react";
-import { Breadcrumb, DefaultPalette, FontSizes, IBreadcrumbItem, IBreadcrumbStyleProps, IBreadcrumbStyles, Image, IStyleFunctionOrObject, Link, SearchBox, Stack, Toggle } from "@fluentui/react";
+import { Breadcrumb, ContextualMenuItemType, DefaultPalette, FontSizes, IBreadcrumbItem, IBreadcrumbStyleProps, IBreadcrumbStyles, IconButton, IContextualMenuProps, IIconProps, Image, IStyleFunctionOrObject, Link, SearchBox, Stack, Toggle } from "@fluentui/react";
 
 import nlsHPCC from "src/nlsHPCC";
 
@@ -27,6 +27,35 @@ export const DevTitle: React.FunctionComponent<DevTitleProps> = ({
         return retVal;
     })];
 
+    const advMenuProps: IContextualMenuProps = {
+        items: [
+            { key: "errors", href: "#/errors", text: nlsHPCC.ErrorWarnings, },
+            { key: "divider_1", itemType: ContextualMenuItemType.Divider },
+            { key: "banner", text: nlsHPCC.SetBanner },
+            { key: "toolbar", text: nlsHPCC.SetToolbar },
+            { key: "divider_2", itemType: ContextualMenuItemType.Divider },
+            { key: "docs", href: "https://hpccsystems.com/training/documentation/", text: nlsHPCC.Documentation, target: "_blank" },
+            { key: "downloads", href: "https://hpccsystems.com/download", text: nlsHPCC.Downloads, target: "_blank" },
+            { key: "releaseNotes", href: "https://hpccsystems.com/download/release-notes", text: nlsHPCC.ReleaseNotes, target: "_blank" },
+            { key: "additionalResources", text: nlsHPCC.AdditionalResources, subMenuProps: {
+                items: [
+                    { key: "redBook", href: "https://wiki.hpccsystems.com/display/hpcc/HPCC+Systems+Red+Book", text: nlsHPCC.RedBook, target: "_blank" },
+                    { key: "forums", href: "https://hpccsystems.com/bb/", text: nlsHPCC.Forums, target: "_blank" },
+                    { key: "issues", href: "https://track.hpccsystems.com/issues/", text: nlsHPCC.IssueReporting, target: "_blank" },
+                ]
+            }},
+            { key: "divider_3", itemType: ContextualMenuItemType.Divider },
+            { key: "lock", text: nlsHPCC.Lock },
+            { key: "logout", text: nlsHPCC.Logout },
+            { key: "divider_4", itemType: ContextualMenuItemType.Divider },
+            { key: "config", text: nlsHPCC.Configuration },
+            { key: "about", text: nlsHPCC.About },
+        ],
+        directionalHintFixed: true
+    };
+
+    const advMenuIconProps: IIconProps = { iconName: "ContextMenu" };
+
     return <>
         <Stack tokens={{ padding: 9, childrenGap: 9 }} >
             <Stack horizontal disableShrink horizontalAlign="space-between">
@@ -51,6 +80,9 @@ export const DevTitle: React.FunctionComponent<DevTitleProps> = ({
                             onChange={() => setUseDarkMode(!useDarkMode)}
                         />
                     </Stack.Item>
+                    <Stack.Item>
+                        <IconButton title="Advanced" ariaLabel="Advanced" menuProps={ advMenuProps } iconProps={ advMenuIconProps } />
+                    </Stack.Item>
                 </Stack>
             </Stack>
         </Stack>

helm/hpcc/templates/_helpers.tpl

@@ -481,6 +481,51 @@ A kludge to ensure mounted storage (e.g. for nfs, minikube or docker for desktop
 {{- end }}
 
 {{/*
+Container to watch for a file on a shared mount and execute a command
+Pass in dict with me and command
+NB: an alternative to sleep loop would be to install and make use of inotifywait
+*/}}
+{{- define "hpcc.addWaitAndRunContainer" -}}
+- name: wait-and-run
+  image: busybox
+  command:
+    - sh
+    - "-c"
+    - |
+      /bin/sh <<'EOSCRIPT'
+      set -e
+      while true; do
+        if [ -f /wait-and-run/{{ .me.name }}.jobdone ]; then break; fi
+        echo waiting for /wait-and-run/{{ .me.name }}.jobdone
+        sleep 5
+      done
+      echo "Running: {{ .command }}"
+      if {{ .command }}; then
+        echo "Command succeeded"
+      fi
+      EOSCRIPT
+  volumeMounts:
+  - name: wait-and-run
+    mountPath: "/wait-and-run"
+{{- end }}
+
+{{/*
+Add wait-and-run shared inter container mount
+*/}}
+{{- define "hpcc.addWaitAndRunVolumeMount" -}}
+- name: wait-and-run
+  mountPath: "/wait-and-run"
+{{- end }}
+
+{{/*
+Add wait-and-run shared inter container volume
+*/}}
+{{- define "hpcc.addWaitAndRunVolume" -}}
+- name: wait-and-run
+  emptyDir: {}
+{{- end }}
+
+{{/*
 Check dll mount point, using hpcc.changeMountPerms
 */}}
 {{- define "hpcc.checkDllMount" -}}
@@ -876,3 +921,30 @@ Pass in dict with root, pod, target and type
 {{- end -}}
 {{- end -}}
 {{- end -}}
+
+{{/*
+Generate lifecycle, command and args
+Pass in root, me and command
+*/}}
+{{- define "hpcc.addCommandAndLifecycle" -}}
+{{- $misc := .root.Values.global.misc | default dict }}
+{{- $postJobCommand := $misc.postJobCommand | default "" }}
+{{- if and (not $misc.postJobCommandViaSidecar) $postJobCommand }}
+lifecycle:
+  preStop:
+    exec:
+      command:
+      - >-
+          {{ $postJobCommand }}
+{{- end }}
+command: ["/bin/bash"]
+args:
+- -c
+- >-
+    {{ .command }}
+{{- if $misc.postJobCommandViaSidecar -}} ;
+    touch /wait-and-run/{{ .me.name }}.jobdone
+{{- else if $postJobCommand -}} ;
+    {{ $postJobCommand }}
+{{- end }}
+{{- end -}}

helm/hpcc/templates/eclagent.yaml

@@ -21,6 +21,7 @@ data:
     global:
 {{ include "hpcc.generateGlobalConfigMap" .root | indent 6 }}
 {{- if not .me.useChildProcesses }}
+{{- $misc := .root.Values.global.misc | default dict }}
   {{ $apptype }}-jobspec.yaml: |
     apiVersion: batch/v1
     kind: Job
@@ -36,41 +37,36 @@ data:
         spec:
           {{- include "hpcc.placementsByJobTargetType" (dict "root" .root "job" $appJobName "target" .me.name "type" "eclagent") | indent 10 }}
           serviceAccountName: "hpcc-default"
+{{- if $misc.postJobCommandViaSidecar }}
+          shareProcessNamespace: true
+{{- end }}            
           containers:
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunContainer" (. | merge (dict "command" $misc.postJobCommand)) | indent 10 }}
+{{- end }}
           - name: {{ $appJobName }}
 {{- include "hpcc.addSecurityContext" . | indent 12 }}
 {{ include "hpcc.addImageAttrs" . | indent 12 }}
 {{- include "hpcc.addResources" (dict "me" .me.resources) | indent 12 }}
-{{- $misc := .root.Values.global.misc | default dict }}
-{{- $postJobCommand := $misc.postJobCommand | default "" }}
 {{- $appCmd := printf "%s %s %s %%args" $apptype (include "hpcc.configArg" .me) (include "hpcc.daliArg" .root) }}
+{{ include "hpcc.addCommandAndLifecycle" (. | merge (dict "command" $appCmd)) | indent 12 }}
             workingDir: /var/lib/HPCCSystems
-{{- if $postJobCommand }}
-            lifecycle:
-              preStop:
-                exec:
-                  command:
-                  - >-
-                      {{ $misc.postJobCommand }}
-{{- end }}
-            command: ["/bin/bash"]
-            args:
-            - -c
-            - >-
-                {{ $appCmd }}
-{{- if $postJobCommand -}} ;
-                {{ $misc.postJobCommand }}
-{{- end }}
             volumeMounts:
 {{ include "hpcc.addConfigMapVolumeMount" .me | indent 12 }}
 {{ include "hpcc.addDataVolumeMount" . | indent 12 }}
 {{ include "hpcc.addDllVolumeMount" . | indent 12 }}
 {{ include "hpcc.addSecretVolumeMounts" . | indent 12 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolumeMount" . | indent 12 }}
+{{- end }}
           volumes:
 {{ include "hpcc.addConfigMapVolume" .me | indent 10 }}
 {{ include "hpcc.addDataVolume" . | indent 10 }}
 {{ include "hpcc.addDllVolume" . | indent 10 }}
 {{ include "hpcc.addSecretVolumes" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolume" . | indent 10 }}
+{{- end }}
           restartPolicy: Never
       backoffLimit: 0
 {{- end }}

helm/hpcc/templates/eclccserver.yaml

@@ -20,6 +20,7 @@ data:
     global:
 {{ include "hpcc.generateGlobalConfigMap" .root | indent 6 }}
 {{- if not .me.useChildProcesses }}
+{{- $misc := .root.Values.global.misc | default dict }}
   compile-jobspec.yaml: |
     apiVersion: batch/v1
     kind: Job
@@ -35,7 +36,13 @@ data:
         spec:
           {{- include "hpcc.placementsByJobTargetType" (dict "root" .root "job" $compileJobName "target" .me.name "type" "eclccserver") | indent 10 }}
           serviceAccountName: "hpcc-default"
+{{- if $misc.postJobCommandViaSidecar }}
+          shareProcessNamespace: true
+{{- end }}            
           containers:
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunContainer" (. | merge (dict "command" $misc.postJobCommand)) | indent 10 }}
+{{- end }}
           - name: {{ $compileJobName }}
 {{- include "hpcc.addSecurityContext" . | indent 12 }}
 {{- include "hpcc.addResources" (dict "me" .me.resources) | indent 12 }}
@@ -43,33 +50,24 @@ data:
 {{- $misc := .root.Values.global.misc | default dict -}}
 {{- $postJobCommand := $misc.postJobCommand | default "" }}
 {{- $eclccserverCmd := printf "eclccserver %s %s %%args" (include "hpcc.configArg" .me) (include "hpcc.daliArg" .root) }}
+{{ include "hpcc.addCommandAndLifecycle" (. | merge (dict "command" $eclccserverCmd)) | indent 12 }}
             workingDir: /tmp
-{{- if $postJobCommand }}
-            lifecycle:
-              preStop:
-                exec:
-                  command:
-                  - >-
-                      {{ $misc.postJobCommand }}
-{{- end }}
-            command: ["/bin/bash"]
-            args:
-            - -c
-            - >-
-                {{ $eclccserverCmd }}
-{{- if $postJobCommand -}} ;
-                {{ $misc.postJobCommand }}
-{{- end }}
             volumeMounts:
 {{ include "hpcc.addConfigMapVolumeMount" .me | indent 12 }}
 {{ include "hpcc.addDataVolumeMount" . | indent 12 }}
 {{ include "hpcc.addDllVolumeMount" . | indent 12 }}
 {{ include "hpcc.addSecretVolumeMounts" . | indent 12 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolumeMount" . | indent 12 }}
+{{- end }}
           volumes:
 {{ include "hpcc.addConfigMapVolume" .me | indent 10 }}
 {{ include "hpcc.addDataVolume" . | indent 10 }}
 {{ include "hpcc.addDllVolume" . | indent 10 }}
 {{ include "hpcc.addSecretVolumes" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolume" . | indent 10 }}
+{{- end }}
           restartPolicy: Never
       backoffLimit: 0
 {{- end }}

helm/hpcc/templates/thor.yaml

@@ -59,39 +59,36 @@ data:
           serviceAccountName: "hpcc-agent"
           initContainers: 
             {{- include "hpcc.checkDataMount" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+          shareProcessNamespace: true
+{{- end }}            
           containers:
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunContainer" (. | merge (dict "command" $misc.postJobCommand)) | indent 10 }}
+{{- end }}
           - name: {{ $eclAgentJobName }}
 {{- include "hpcc.addSecurityContext" . | indent 12 }}
 {{ include "hpcc.addImageAttrs" . | indent 12 }}
 {{- include "hpcc.addResources" (dict "me" .eclAgentResources) | indent 12 }}
 {{- $agentCmd := printf "%s %s %s %%args" $eclAgentType (include "hpcc.configArg" .me) (include "hpcc.daliArg" .root) }}
+{{ include "hpcc.addCommandAndLifecycle" (. | merge (dict "command" $agentCmd)) | indent 12 }}
             workingDir: /var/lib/HPCCSystems
-{{- if $postJobCommand }}
-            lifecycle:
-              preStop:
-                exec:
-                  command:
-                  - >-
-                      {{ $misc.postJobCommand }}
-{{- end }}
-            command: ["/bin/bash"]
-            args:
-            - -c
-            - >-
-                {{ $agentCmd }}
-{{- if $postJobCommand -}} ;
-                {{ $misc.postJobCommand }}
-{{- end }}
             volumeMounts:
 {{ include "hpcc.addConfigMapVolumeMount" .me | indent 12 }}
 {{ include "hpcc.addDataVolumeMount" . | indent 12 }}
 {{ include "hpcc.addDllVolumeMount" . | indent 12 }}
 {{ include "hpcc.addSecretVolumeMounts" . | indent 12 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolumeMount" . | indent 12 }}
+{{- end }}
           volumes:
 {{ include "hpcc.addConfigMapVolume" .me | indent 10 }}
 {{ include "hpcc.addDataVolume" . | indent 10 }}
 {{ include "hpcc.addDllVolume" . | indent 10 }}
 {{ include "hpcc.addSecretVolumes" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolume" . | indent 10 }}
+{{- end }}
           restartPolicy: Never
       backoffLimit: 0
 {{- end }}
@@ -115,39 +112,36 @@ data:
           serviceAccountName: hpcc-agent
           initContainers:
             {{- include "hpcc.checkDataMount" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+          shareProcessNamespace: true
+{{- end }}            
           containers:
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunContainer" (. | merge (dict "command" $misc.postJobCommand)) | indent 10 }}
+{{- end }}
           - name: {{ $thorManagerJobName }}
 {{- include "hpcc.addSecurityContext" . | indent 12 }}
 {{ include "hpcc.addImageAttrs" . | indent 12 }}
 {{- include "hpcc.addResources" (dict "me" $thorScope.managerResources) | indent 12 }}
 {{- $thorManagerCmd := printf "thormaster_lcr %s %s %%args" (include "hpcc.configArg" .me) (include "hpcc.daliArg" .root) }}
+{{ include "hpcc.addCommandAndLifecycle" (. | merge (dict "command" $thorManagerCmd)) | indent 12 }}
             workingDir: /var/lib/HPCCSystems
-{{- if $postJobCommand }}
-            lifecycle:
-              preStop:
-                exec:
-                  command:
-                  - >-
-                      {{ $misc.postJobCommand }}
-{{- end }}
-            command: ["/bin/bash"]
-            args:
-            - -c
-            - >-
-                {{ $thorManagerCmd }}
-{{- if $postJobCommand -}} ;
-                {{ $misc.postJobCommand }}
-{{- end }}
             volumeMounts:
 {{ include "hpcc.addConfigMapVolumeMount" .me | indent 12 }}
 {{ include "hpcc.addDataVolumeMount" . | indent 12 }}
 {{ include "hpcc.addDllVolumeMount" . | indent 12 }}
 {{ include "hpcc.addSecretVolumeMounts" . | indent 12 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolumeMount" . | indent 12 }}
+{{- end }}
           volumes:
 {{ include "hpcc.addConfigMapVolume" .me | indent 10 }}
 {{ include "hpcc.addDataVolume" . | indent 10 }}
 {{ include "hpcc.addDllVolume" . | indent 10 }}
 {{ include "hpcc.addSecretVolumes" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolume" . | indent 10 }}
+{{- end }}
           restartPolicy: Never
       backoffLimit: 0
 
@@ -168,39 +162,36 @@ data:
         spec:
           {{- include "hpcc.placementsByJobTargetType" (dict "root" .root "job" $thorWorkerJobName "target" .me.name "type" "thor") | indent 10 }}
           serviceAccountName: hpcc-default
+{{- if $misc.postJobCommandViaSidecar }}
+          shareProcessNamespace: true
+{{- end }}            
           containers:
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunContainer" (. | merge (dict "command" $misc.postJobCommand)) | indent 10 }}
+{{- end }}
           - name: {{ $thorWorkerJobName }}
 {{- include "hpcc.addSecurityContext" . | indent 12 }}
 {{ include "hpcc.addImageAttrs" . | indent 12 }}
 {{- include "hpcc.addResources" (dict "me" $thorScope.workerResources) | indent 12 }}
 {{- $thorWorkerCmd := printf "thorslave_lcr %s %s %%args" (include "hpcc.configArg" .me) (include "hpcc.daliArg" .root) }}
+{{ include "hpcc.addCommandAndLifecycle" (. | merge (dict "command" $thorWorkerCmd)) | indent 12 }}
             workingDir: /var/lib/HPCCSystems
-{{- if $postJobCommand }}
-            lifecycle:
-              preStop:
-                exec:
-                  command:
-                  - >-
-                      {{ $misc.postJobCommand }}
-{{- end }}
-            command: ["/bin/bash"]
-            args:
-            - -c
-            - >-
-                {{ $thorWorkerCmd }}
-{{- if $postJobCommand -}} ;
-                {{ $misc.postJobCommand }}
-{{- end }}
             volumeMounts:
 {{ include "hpcc.addConfigMapVolumeMount" .me | indent 12 }}
 {{ include "hpcc.addDataVolumeMount" . | indent 12 }}
 {{ include "hpcc.addDllVolumeMount" . | indent 12 }}
 {{ include "hpcc.addSecretVolumeMounts" . | indent 12 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolumeMount" . | indent 12 }}
+{{- end }}
           volumes:
 {{ include "hpcc.addConfigMapVolume" .me | indent 10 }}
 {{ include "hpcc.addDataVolume" . | indent 10 }}
 {{ include "hpcc.addDllVolume" . | indent 10 }}
 {{ include "hpcc.addSecretVolumes" . | indent 10 }}
+{{- if $misc.postJobCommandViaSidecar }}
+{{ include "hpcc.addWaitAndRunVolume" . | indent 10 }}
+{{- end }}
           restartPolicy: Never
       backoffLimit: 0
 

helm/hpcc/values.schema.json

@@ -220,6 +220,10 @@
                 "postJobCommand": {
                   "description": "Execute this command when a K8s Job launched by a deployment ends, either gracefully or when terminated externally causing a preStop event",
                   "type": "string"
+                },
+                "postJobCommandViaSidecar": {
+                  "description": "Causes shareProcessNamespace to be enabled, and the postJobCommand to execute via a sidecar",
+                  "type": "boolean"
                 }
               }
             },

helm/hpcc/values.yaml

@@ -45,9 +45,16 @@ global:
   # This can be useful if injected sidecars are installed that need to be told to stop.
   # If they are not stopped, the pod continues running with the side car container only, in a "NotReady" state.
   # An example of this is the Istio envoy side car. It can be stopped with the command below.
+  # Set postJobCommandViaSidecar to true, if the command needs to run with priviledge, this will enable the command
+  # to run as root in a sidecar in same process space as other containers, allowing it to for example send signals
+  # to processes in sidecars
 
   # misc:
   #   postJobCommand: "curl -sf -XPOST http://127.0.0.1:15020/quitquitquit"
+  # Or example for linkerd
+  #   postJobCommand: "kill $(pgrep linkerd2-proxy)"
+  #   postJobCommandViaSidecar: true
+
 
   # For pod placement instruction and examples please reference docs/placements.md
 

plugins/couchbase/README.md

@@ -52,6 +52,8 @@ For example:
 
 ##N1QL Query Examples
 
+Note that only a single N1QL statement embedded function is supported at this time.
+
                 //Creates a primary index on 'mybucket' bucket
                 createprimeindex() := EMBED(couchbase : server(server), port(port), bucket('mybucket'))
                     CREATE PRIMARY INDEX ON iot;

plugins/couchbase/couchbaseembed.cpp

@@ -181,6 +181,8 @@ namespace couchbaseembed
             auto status = pQcmd->named_param(cbPlaceholder.str(), utf8);
             if (!status.success())
                 failx("Could not bind Param: %s val: %s", cbPlaceholder.str(), utf8);
+            if (utf8)
+                rtlFree(utf8);
         }
         else
             failx("Internal error: detected invalid CouchbaseQueryCommand while attempting to bind to field: %s", cbPlaceholder.str());
@@ -215,6 +217,8 @@ namespace couchbaseembed
             auto status = pQcmd->named_param(cbPlaceholder.str(), (char *)data);
             if (!status.success())
                 failx("Could not bind Param: %s val: %s", cbPlaceholder.str(), (char *)data);
+            if (data)
+                rtlFree(data);
         }
         else
             failx("Internal error: detected invalid CouchbaseQueryCommand while attempting to bind to field: %s", cbPlaceholder.str());
@@ -912,32 +916,33 @@ namespace couchbaseembed
 
     IPropertyTree * CouchbaseEmbedFunctionContext::nextResultRowTree()
     {
-        for (auto cbrow : *m_pQuery)
+        if (m_pQuery)
         {
-            auto json = cbrow.json().to_string();
+            reportIfQueryFailure(m_pQuery);
+
+            // Only the first callback query is processed
+            auto json = m_pQuery->begin()->json().to_string();
             Owned<IPropertyTree> contentTree = createPTreeFromJSONString(json.c_str());
             return contentTree.getLink();
         }
 
-        reportIfQueryFailure(m_pQuery);
-
         return nullptr;
     }
 
     IPropertyTreeIterator * CouchbaseEmbedFunctionContext::nextResultRowIterator()
     {
-        for (auto cbrow : *m_pQuery)
+        if (m_pQuery)
         {
-            auto json = cbrow.json().to_string();
+            reportIfQueryFailure(m_pQuery);
+
+            // Only the first callback query is processed
+            auto json = m_pQuery->begin()->json().to_string();
             Owned<IPropertyTree> contentTree = createPTreeFromJSONString(json.c_str());
             if (contentTree)
                 return contentTree->getElements("./*");
             failx("Could not fetch next result row.");
-            break;
         }
 
-        reportIfQueryFailure(m_pQuery);
-
         return nullptr;
     }
 
@@ -1126,6 +1131,8 @@ namespace couchbaseembed
         auto status = m_pQcmd->named_param(cbPlaceholder.str(), (char *)data);
         if (!status.success())
             failx("Could not bind Param: %s val: %s", cbPlaceholder.str(), (char *)data);
+        if (data)
+            rtlFree(data);
     }
 
     void CouchbaseEmbedFunctionContext::bindFloatParam(const char *name, float val)
@@ -1198,6 +1205,8 @@ namespace couchbaseembed
         auto status = m_pQcmd->named_param(cbPlaceholder.str(), utf8);
         if (!status.success())
             failx("Could not bind Param: %s val: %s", cbPlaceholder.str(), utf8);
+        if (utf8)
+            rtlFree(utf8);
     }
 
     void CouchbaseEmbedFunctionContext::bindVStringParam(const char *name, const char *val)
@@ -1404,11 +1413,8 @@ namespace couchbaseembed
         size32_t chars;
         rtlDataAttr result;
         value.setString(strlen(dvalue), dvalue);
-        if (field)
-        {
-            RtlDecimalTypeInfo *dtype = (RtlDecimalTypeInfo *) field->type;
-            value.setPrecision(dtype->getDecimalDigits(), dtype->getDecimalPrecision());
-        }
+        RtlDecimalTypeInfo *dtype = (RtlDecimalTypeInfo *) field->type;
+        value.setPrecision(dtype->getDecimalDigits(), dtype->getDecimalPrecision());
     }
 
     void CouchbaseRowBuilder::processBeginSet(const RtlFieldInfo * field, bool &isAll)