Merge pull request #13827 from richardkchapman/unified-image

HPCC-24232 Use common image and document how to add 3rd-party components

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>

dockerfiles/README.md

@@ -3,43 +3,35 @@ Docker images
 
 Docker images related to HPCC are structured as follows
 
-hpccsystems/platform-build-base
+**hpccsystems/platform-build-base**
 
 This image contains all the development packages required to build the hpcc platform,
-but no HPCC code or sources. It changes rarely. The current version is tagged 7.8 and
-is based on Ubuntu 18.04 base image
+but no HPCC code or sources. It changes rarely. The current version is tagged 7.10 and
+is based on Ubuntu 20.04 base image
 
 **hpccsystems/platform-build**
 
-Building this image builds an installation package (.deb file) for a specified git tag
-of the HPCC platform sources. The Dockerfile takes two arguments, naming the version of
-the platform-build-base image to use, and the git tag to use. Sources are fetched from
+Building this image builds and installs the HPCC codebase for a specified git tag
+of the HPCC platform sources. The Dockerfile takes arguments naming the version of
+the platform-build-base image to use and the git tag to use. Sources are fetched from
 github. An image will be pushed to Dockerhub for every public tag on the HPCC-Platform
 repository in GitHub, which developers can use as a base for their own development.
 
-There is a second Dockerfile inplatform-build-incremental that can be used by developers
+There is a second Dockerfile in platform-build-incremental that can be used by developers
 working on a branch that is not yet tagged or merged into upstream, that uses 
 hpccsystems/platform-build as a base in order to avoid the need for full rebuilds each time
 the image is built.
 
 **hpccsystems/plaform-core**
 
-This uses the .deb file from a hpccsystems/plaform-build image to install a copy of the
-full platform code, without specialization to a specific component.
+This uses the build artefacts file from a hpccsystems/plaform-build image to install a copy of the
+full platform code, which can be used to run any HPCC component.
 
-**hpccsystems/dali**  
-**hpccsystems/roxie**  
-**hpccsystems/esp**  
-**etc**  
+If you need additional components installed on your cluster, such as Python libraries, you can create
+a Docker image based on platform-core with additional components installed. An example can be found
+in the examples/numpy directory. You will then override the image name when deploying a helm chart in
+order to enabled your additional components.
 
-These are specializations of the platform-core image to run a specific component.
-Portions of the platform-core that are not needed by this component may be removed.
-These images are the ones that are referred to in helm scripts etc when launching
-a cloud cluster.
-
-If launched without further parameters or configuration, a system with default
-settings can be started, but it will be more normal to apply some configuration at
-container launch time.
 
 ---
 Bash scripts
@@ -55,7 +47,7 @@ stopall.sh  - Stop a local k8s cluster
 Helm chart
 ==========
 
-The Helm chart in hpcc/ can be used to deploy an entire HPCC environment to a K8s cluster.
+The Helm chart in helm/hpcc/ can be used to deploy an entire HPCC environment to a K8s cluster.
 
 values.yaml sections
 --------------------

dockerfiles/buildall.sh

@@ -99,15 +99,6 @@ build_image() {
 build_image platform-build-base ${BASE_VER}
 build_image platform-build
 build_image platform-core
-build_image roxie
-build_image dali
-build_image esp
-build_image eclccserver
-build_image eclagent
-build_image hthor
-build_image toposerver
-build_image thormaster
-build_image thorslave
 
 if [[ -n ${INPUT_PASSWORD} ]] ; then
   echo "::set-output name=${BUILD_LABEL}"

dockerfiles/eclagent/Dockerfile

@@ -1,27 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single agentexec server
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/eclagent
-WORKDIR /var/lib/HPCCSystems/eclagent
-ENTRYPOINT ["agentexec"]

dockerfiles/eclccserver/Dockerfile

@@ -1,27 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single eclcccserver server
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/eclccserver
-WORKDIR /var/lib/HPCCSystems/eclccserver
-ENTRYPOINT [ "eclccserver" ]

dockerfiles/esp/Dockerfile

@@ -1,30 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single ESP server
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/esp
-WORKDIR /var/lib/HPCCSystems/esp
-COPY --chown=hpcc:hpcc esp.xml /var/lib/HPCCSystems/esp
-#RUN esp --init
-ENTRYPOINT ["esp"]
-

dockerfiles/dali/Dockerfile

@@ -15,15 +15,16 @@
 #    limitations under the License.
 ##############################################################################
 
-# Build container image for a single dali server
+# Build container image for HPCC components including numpy
 
 ARG BUILD_LABEL
-ARG DOCKER_REPO
+ARG DOCKER_REPO=hpccsystems
 FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
 
-USER hpcc
+USER root
+RUN apt install -y python3-pip
 
-RUN mkdir -p /var/lib/HPCCSystems/dali
-WORKDIR /var/lib/HPCCSystems/dali
-ENTRYPOINT ["daserver"]
+USER hpcc
+RUN pip3 install numpy
+RUN pip3 install tensorflow
 

dockerfiles/examples/numpy/build.sh

@@ -0,0 +1,8 @@
+LABEL=$1
+[[ -n ${LABEL} ]] && LABEL=latest
+
+docker image build -t hpccsystems/platform-core:numpy \
+     --build-arg DOCKER_REPO=hpccsystems \
+     --build-arg BUILD_LABEL=${LABEL} \
+     addnumpy/ 
+

dockerfiles/hthor/Dockerfile

@@ -1,27 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single hthor process
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/hthor
-WORKDIR /var/lib/HPCCSystems/hthor
-ENTRYPOINT ["hthor"]

dockerfiles/incr.sh

@@ -162,15 +162,6 @@ else
   echo BUILDING: build_image platform-core platform-core-debug
   build_image platform-core platform-core-debug
 fi
-build_image roxie
-build_image dali
-build_image esp
-build_image eclccserver
-build_image eclagent
-build_image hthor
-build_image toposerver
-build_image thormaster
-build_image thorslave
 
 echo Built ${DOCKER_REPO}/*:${BUILD_LABEL}
 

dockerfiles/platform-build-incremental/Dockerfile

@@ -56,5 +56,6 @@ RUN echo Building with $(cat ~/build_threads) threads
 RUN make -j$(cat ~/build_threads)
 
 USER root
+RUN cmake /hpcc-dev/HPCC-Platform -DUSE_PYTHON2=0
 RUN make -j$(cat ~hpcc/build_threads) install
 

dockerfiles/platform-core-debug/Dockerfile

@@ -36,3 +36,5 @@ ENV HPCC_containerized=1
 ENV HPCC_DLLSERVER_PATH=/var/lib/HPCCSystems/queries
 
 USER hpcc
+# temporary workaround for ESP configuration not being implemented yet
+COPY --chown=hpcc:hpcc esp.xml /var/lib/HPCCSystems

dockerfiles/platform-core/Dockerfile

@@ -84,3 +84,6 @@ USER hpcc
 ENV PATH="/opt/HPCCSystems/bin:${PATH}"
 ENV HPCC_containerized=1
 ENV HPCC_DLLSERVER_PATH=/var/lib/HPCCSystems/queries
+# temporary workaround for ESP configuration not being implemented yet
+COPY --chown=hpcc:hpcc esp.xml /var/lib/HPCCSystems
+

dockerfiles/roxie/Dockerfile

@@ -1,28 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single roxie process
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/roxie
-WORKDIR /var/lib/HPCCSystems/roxie
-ENTRYPOINT ["roxie"]
-

dockerfiles/thormaster/Dockerfile

@@ -1,28 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single thorslave
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/thor
-WORKDIR /var/lib/HPCCSystems/thor
-ENTRYPOINT ["thormaster_lcr"]
-

dockerfiles/thorslave/Dockerfile

@@ -1,28 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a single thormaster
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/thor
-WORKDIR /var/lib/HPCCSystems/thor
-ENTRYPOINT ["thorslave_lcr"]
-

dockerfiles/toposerver/Dockerfile

@@ -1,29 +0,0 @@
-##############################################################################
-#
-#    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-##############################################################################
-
-# Build container image for a roxie topologyserver process
-
-ARG BUILD_LABEL
-ARG DOCKER_REPO
-FROM ${DOCKER_REPO}/platform-core:${BUILD_LABEL}
-
-USER hpcc
-RUN mkdir -p /var/lib/HPCCSystems/toposerver
-WORKDIR /var/lib/HPCCSystems/toposerver
-RUN toposerver --init > toposerver.yaml
-ENTRYPOINT ["toposerver", "--config=toposerver.yaml"]
-

helm/hpcc/templates/_helpers.tpl

@@ -146,11 +146,11 @@ Add dali arg for a component
 Get image name
 */}}
 {{- define "hpcc.imageName" -}}
-{{- /* Pass in a dictionary with root, me and imagename defined */ -}}
+{{- /* Pass in a dictionary with root and me defined */ -}}
 {{- if .me.image -}}
-{{ .me.image.root | default .root.Values.global.image.root | default "hpccsystems" }}/{{ .imagename }}:{{ .me.image.version | default .root.Values.global.image.version }}
+{{ .me.image.root | default .root.Values.global.image.root | default "hpccsystems" }}/{{ .me.image.name | default .root.Values.global.image.name | default "platform-core" }}:{{ .me.image.version | default .root.Values.global.image.version }}
 {{- else -}}
-{{ .root.Values.global.image.root | default "hpccsystems" }}/{{ .imagename }}:{{ .root.Values.global.image.version }}
+{{ .root.Values.global.image.root | default "hpccsystems" }}/{{ .root.Values.global.image.name | default "platform-core" }}:{{ .root.Values.global.image.version }}
 {{- end -}}
 {{- end -}}
 

helm/hpcc/templates/dali.yaml

@@ -17,6 +17,8 @@ spec:
       {{- include "hpcc.checkDaliMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ .name | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ daserver ] 
         args: [
                 {{ include "hpcc.configArg" . }}
               ]

helm/hpcc/templates/eclagent.yaml

@@ -19,6 +19,8 @@ spec:
       {{- include "hpcc.checkDataMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ .name | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ agentexec ] 
         args: [ 
                 {{ include "hpcc.configArg" . }},
                 {{ include "hpcc.daliArg" $ }}
@@ -69,8 +71,9 @@ data:
               limits:
                 #limit-memory
                 #limit-cpu
-            command: [
-                        {{ $apptype | quote }},
+            workingDir: /var/lib/HPCCSystems
+            command: [ {{ $apptype | quote }} ]
+            args: [
                         {{ include "hpcc.configArg" . }},
                         {{ include "hpcc.daliArg" $ }},
                         %args

helm/hpcc/templates/eclccserver.yaml

@@ -19,6 +19,8 @@ spec:
       {{- include "hpcc.checkDllMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ .name | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ eclccserver ] 
         args: [
                 {{ include "hpcc.configArg" . }},
                 {{ include "hpcc.daliArg" $ }}
@@ -68,7 +70,9 @@ data:
               limits:
                 #limit-memory
                 #limit-cpu
-            command: [
+            workingDir: /var/lib/HPCCSystems
+            command: [ eclccserver ]
+            args: [
                         "eclccserver",
                         {{ include "hpcc.configArg" . }},
                         {{ include "hpcc.daliArg" $ }},

helm/hpcc/templates/esp.yaml

@@ -16,6 +16,8 @@ spec:
     spec:
       containers:
       - name: {{ .name | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ esp ] 
         args: [
                 {{ include "hpcc.configArg" . }},
                 {{ include "hpcc.daliArg" $ }}

helm/hpcc/templates/localroxie.yaml

@@ -22,6 +22,8 @@ spec:
       {{- include "hpcc.checkDataMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ $roxie.name | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ roxie ]
         args: [
                 {{ include "hpcc.configArg" $roxie }},
                 {{ include "hpcc.daliArg" $ }},

helm/hpcc/templates/roxie.yaml

@@ -24,6 +24,8 @@ spec:
       - name: {{ $toponame | quote }}
 {{- include "hpcc.addSecurityContext" (dict "root" $ "me" .) | indent 8 }}
 {{ include "hpcc.addImageAttrs" (dict "root" $ "me" $roxie "imagename" "toposerver") | indent 8 }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ toposerver ] 
         volumeMounts:
         - name: {{ .name }}-configmap-volume
           mountPath: /etc/config
@@ -100,6 +102,8 @@ spec:
       {{- include "hpcc.checkDataMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ $servername | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ roxie ] 
         args: [
                 {{ include "hpcc.configArg" . }},
                 {{ include "hpcc.daliArg" $ }},
@@ -156,6 +160,8 @@ spec:
       {{- include "hpcc.checkDataMount" (dict "root" $) | indent 6 }}
       containers:
       - name: {{ $name | quote}}
+        workingDir: /var/lib/HPCCSystems
+        command: [ roxie ]
         args: [ 
                 {{ include "hpcc.configArg" $roxie }},
                 {{ include "hpcc.daliArg" $ }},

helm/hpcc/templates/thor.yaml

@@ -31,6 +31,8 @@ spec:
       serviceAccountName: hpcc
       containers:
       - name: {{ $eclAgentName | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ agentexec ]
         args: [
                 {{ include "hpcc.configArg" . }},
                 {{ include "hpcc.daliArg" $ }},
@@ -64,6 +66,8 @@ spec:
       serviceAccountName: hpcc
       containers:
       - name: {{ $thorAgentName | quote }}
+        workingDir: /var/lib/HPCCSystems
+        command: [ agentexec ]
         args: [
                 {{ include "hpcc.configArg" $thorAgentScope }},
                 {{ include "hpcc.daliArg" $ }},
@@ -142,8 +146,9 @@ data:
               limits:
                 #limit-memory
                 #limit-cpu
-            command: [
-                        {{ $agentAppType | quote }},
+            workingDir: /var/lib/HPCCSystems
+            command: [ {{ $agentAppType | quote }} ] 
+            args: [
                         {{ include "hpcc.configArg" . }},
                         {{ include "hpcc.daliArg" $ }},
                         %args
@@ -176,6 +181,8 @@ data:
           - name: %jobname
 {{- include "hpcc.addSecurityContext" (dict "root" $ "me" .) | indent 12 }}
 {{ include "hpcc.addImageAttrs" (dict "root" $ "me" . "imagename" "thormaster") | indent 12 }}
+            workingDir: /var/lib/HPCCSystems
+            command: [ thormaster_lcr ] 
             args: [
                     {{ include "hpcc.configArg" . }},
                     {{ include "hpcc.daliArg" $ }},
@@ -208,6 +215,8 @@ data:
           - name: %jobname
 {{- include "hpcc.addSecurityContext" (dict "root" $ "me" .) | indent 12 }}
 {{ include "hpcc.addImageAttrs" (dict "root" $ "me" . "imagename" "thorslave") | indent 12 }}
+            workingDir: /var/lib/HPCCSystems
+            command: [ thorslave_lcr ] 
             args: [
                     {{ include "hpcc.configArg" . }},
                     {{ include "hpcc.daliArg" $ }},