HPCC-25357 Move Elastic dependency out of root HPCC chart

- Removes root dependency on Elastic components
- Updates README.md
- Creates helm/managed directory and subdirectory
- Creates new README.md for new directories
- Adds NOTES template, discloses PVC persistance to user

Signed-off-by: Rodrigo Pastrana <Rodrigo.Pastrana@lexisnexisrisk.com>

helm/README.md

@@ -1,33 +0,0 @@
-## HPCC Systems HELM Chart
-
-### Dependencies
-
-#### Elastic Stack
-HPCC Systems is designed to deploy a local, minimal Elastic Stack instance for log processing out of the box. This feature creates a local dependency on the Elastic Stack Helm charts for ElasticSearch, Filebeats and optionally Kibana.
-
-##### HELM Command
-Helm provides a convenient command to automatically pull appropriate dependencies to the /charts directory:
-> helm dependency update HPCC-Platform/helm/hpcc
-
-Note: *HPCC-Platform/helm/hpcc* denotes the location of the HPCC Platform HELM chart
-##### HELM Install parameter
-Otherwise, provide the "--dependency-update" argument in the helm install command
-For example:
-> helm install myhpcccluster HPCC-Platform/helm/hpcc  --set global.image.version=latest --dependency-update
-
-##### Disabling
-The managed Elastic Stack instance can be disabled by setting the following values to false:
->--set elasticsearch.enabled=false
-and
->--set kibana.enabled=false
-
-For example:
-> helm install myhpcccluster HPCC-Platform/helm/hpcc  --set global.image.version=latest --set elasticsearch.enabled=false --set kibana.enabled=false
-
-However, the HELM install command verifies that the required charts, as expressed in Chart.yaml, are present in charts/ and are at an acceptable version even when above dependency flags are disabled.
-
-##### Removing
-If the user chooses to avoid the Elastic Stack dependency altogether, the depedencies declaration in the Chart.yml can be removed. Please note, absent of the Elastic Stack instance, the user would be resposible for HPCC component log processing and persitance strategies. Please see for more information /HPCC-Platform/helm/examples/logging/README.md
-
-##### Cleanup
-The Elasticsearch chart will declare a PVC which is used to persist data related to its indexes, and thus, the HPCC Component logs. The PVCs by nature can outlive the HPCC and Elastic deployments, it is up to the user to manage the PVC appropriately, which includes deleting the PVC when they are no longer needed.

helm/examples/logging/elastic/README.md

@@ -1,8 +1,13 @@
 # HPCC Log Processing via Elastic Stack
 
+## HPCC Managed Lightweight Elastic Stack
+HPCC provides a simple chart for deployment of a local, light-weight Elastic Stack instance.
+Please view HPCC-Platform/helm/managed/logging/elastic for more information
+
+## Stand Alone Elastic Stack
 Setting up a base Elastic Stack cluster to process HPCC Systems component logs is straightforward. Elastic provides Helm charts to deploy each of their components, so we'll add the Elastic helm-charts repository locally:
 
-	> helm repo add elastic https://helm.elastic.co
+    > helm repo add elastic https://helm.elastic.co
 
 We'll install the Filebeat component (log agent) , and ElasticSearch (log store and indexer). By default, Filebeat will forward the log entries to the ElasticSearch default endpoint:
 

helm/hpcc/Chart.yaml

@@ -12,27 +12,3 @@ version: 0.1.0
 # incremented each time you make changes to the application.
 
 appVersion: 0.1.0
-
-# Dependencies can be automatically updated via HELM dependancy update command:
-# > 'helm dependency update HPCC-Systems/helm/hpcc' command
-# or helm install parameter —dependency-update
-# > helm install HPCC-Systems/helm/hpcc --set global.image.version=latest —-dependency-update
-# For details please see HPCC-Systems/helm/hpcc/README.md
-dependencies:
-# An Elastic Stack environment is managed by default
-# To disable: --set elasticsearch.enabled=false --set kibana.enabled=false
-# and speficy alternate log processor
-# Kibana is an optional front-end system, and is disabled by default.
-# To Enable: --set elasticsearch.enabled=true --set kibana.enabled=true
-- name: filebeat # Managed stderr log-forwarder
-  version: 7.9.3
-  repository: https://helm.elastic.co
-  condition: elasticsearch.enabled
-- name: elasticsearch # Managed logging processor engine
-  version: 7.9.3
-  repository: https://helm.elastic.co
-  condition: elasticsearch.enabled
-- name: kibana # Optional managed logging processor front-end
-  version: 7.9.3
-  repository: https://helm.elastic.co
-  condition: kibana.enabled #should only enable if elasticsearch.enabled=true

helm/hpcc/values.schema.json

@@ -154,15 +154,6 @@
       "description": "sasha services",
       "type": "object",
       "$ref": "#/definitions/sashaservice"
-    },
-    "kibana": {
-      "description": "HPCC managed Elastic Stack deployment Kibana (log visualizer) component"
-    },
-    "elasticsearch": {
-      "description": "HPCC managed Elastic Stack deployment ElasticSearch (log processor) component"
-    },
-    "filebeat": {
-      "description": "HPCC managed Elastic Stack deployment Filebeat (log agent) component"
     }
   },
   "required": [

helm/hpcc/values.yaml

@@ -49,69 +49,6 @@ security:
     extern: "allow"
     datafile: "allow"
 
-##Minimal instance of Elastic stack will be deployed alongside HPCC components
-##Elastic stack is comprised of several components which follow below and can be customized
-##by overwriting/setting helm chart values here
-
-##The elasticsearch component can be customized by modifying helm chart values here.
-elasticsearch:
-  enabled: true
-  description: "HPCC Managed Elasticsearch"
-  ##See https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml for all available options
-  antiAffinity: "soft"  #default is HARD, for minimal systems soft might be necessary
-  replicas: 1           #default is 3, for minimal systems 1 replicas should be adequate
-  minimumMasterNodes: 1 #default is 2, for minimal systems 1 master node should be adequate
-  labels: {"managedby" : "HPCC"}
-  clusterHealthCheckParams: "local=true" #local node health status
-  #volumeClaimTemplate:
-    #accessModes: [ "ReadWriteOnce" ]
-    #resources:
-    #requests:
-    #  storage: 30Gi
-  #persistence:
-    #enabled: true
-    #labels:
-      # Add default labels for the volumeClaimTemplate fo the StatefulSet
-      #enabled: false
-    #annotations: {}
-
-##The filebeat component can be customized by modifying helm chart values here.
-filebeat:
-  description: "HPCC Managed filebeat"
-  ##See https://github.com/elastic/helm-charts/blob/master/filebeat/values.yaml for all available options
-  labels: {"managedby" : "HPCC"}
-  ## Allows you to add any config files in /usr/share/filebeat
-  ## such as filebeat.yml
-  #filebeatConfig:
-    #filebeat.yml: |
-      #filebeat.inputs:
-      #- type: container
-      #  paths:
-      #    - /var/log/containers/*.log
-        #processors:
-        #- add_kubernetes_metadata:
-            #host: ${NODE_NAME}
-            #matchers:
-            #- logs_path:
-                #logs_path: "/var/log/containers/"
-      #output.elasticsearch:
-      #  host: '${NODE_NAME}'
-      #  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
-      #  index: "hpcc-%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
-
-##The kibana component can be customized by modifying helm chart values here.
-kibana:
-  enabled: false
-  description: "HPCC Managed Kibana"
-  ##See https://github.com/elastic/helm-charts/blob/master/kibana/values.yaml for all available options
-  labels: {"managedby" : "HPCC"}
-  ## Allows you to add any config files in /usr/share/kibana/config/
-  ## such as kibana.yml
-  #kibanaConfig: {}
-  #   kibana.yml: |
-  #     key:
-  #       nestedkey: value
-
 ## storage:
 ##
 ## If storage.[type].existingClaim is defined, a Persistent Volume Claim must

helm/managed/README.md

@@ -0,0 +1,2 @@
+## HPCC Systems Managed Helm Charts
+This directory contains external helm charts and HPCC managed default values

helm/managed/logging/README.md

@@ -0,0 +1,2 @@
+## HPCC Systems Managed Helm Charts
+This directory contains external helm charts and HPCC managed default values related to logging

helm/managed/logging/elastic/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: elastic4hpcclogs
+description: A Helm chart for launching a lightweight Elastic Stack cluster using Kubernetes
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+version: 0.1.0
+
+# Dependencies can be automatically updated via HELM dependancy update command:
+# > 'helm dependency update' command
+# or helm install parameter —dependency-update
+# > helm install myelastic ./ —-dependency-update
+dependencies:
+- name: filebeat
+  version: 7.9.3
+  repository: https://helm.elastic.co
+- name: elasticsearch
+  version: 7.9.3
+  repository: https://helm.elastic.co
+- name: kibana # Optional managed logging processor front-end
+  version: 7.9.3
+  repository: https://helm.elastic.co
+  condition: kibana.enabled

helm/managed/logging/elastic/README.md

@@ -0,0 +1,19 @@
+## This folder contains lightweigth Elastic Stack deployment charts and HPCC prefered values
+
+This chart describes a local, minimal Elastic Stack instance for HPCC Systems component log processing.
+
+### Dependencies
+This chart is dependent on the Elastic Stack Helm charts for ElasticSearch, Filebeats and Kibana.
+
+#### Dependency update
+##### HELM Command
+Helm provides a convenient command to automatically pull appropriate dependencies to the /charts directory:
+> helm dependency update
+
+##### HELM Install parameter
+Otherwise, provide the "--dependency-update" argument in the helm install command
+For example:
+> helm install myelastic ./ --dependency-update
+
+##### Cleanup
+The Elastic Search chart will declare a PVC which is used to persist data related to its indexes, and thus, the HPCC Component logs. The PVCs by nature can outlive the HPCC and Elastic deployments, it is up to the user to manage the PVC appropriately, which includes deleting the PVC when they are no longer needed.

helm/managed/logging/elastic/templates/NOTES.txt

@@ -0,0 +1,6 @@
+Thank you for installing {{ .Chart.Name }}.
+A lightweight Elastic Search instance for HPCC component log processing.
+
+This deployment varies slightly from defaults set by Elastic, please review the effective values.
+
+PLEASE NOTE: Elastic Search declares PVC(s) which might require explicit manual removal when no longer needed.

helm/managed/logging/elastic/values.yaml

@@ -0,0 +1,60 @@
+# Default values for a lightweight Elastic Stack instance which can process HPCC component logs.
+
+##The elasticsearch component can be customized by modifying helm chart values here.
+elasticsearch:
+  enabled: true
+  description: "HPCC Managed Elasticsearch"
+  ##See https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml for all available options
+  antiAffinity: "soft"  #default is HARD, for minimal systems soft might be necessary
+  replicas: 1           #default is 3, for minimal systems 1 replicas should be adequate
+  minimumMasterNodes: 1 #default is 2, for minimal systems 1 master node should be adequate
+  labels: {"managedby" : "HPCC"}
+  clusterHealthCheckParams: "local=true" #local node health status
+  volumeClaimTemplate:
+    accessModes: [ "ReadWriteOnce" ]
+    resources:
+      requests:
+        storage: 5Gi
+  #persistence:
+    #enabled: true
+    #labels:
+      # Add default labels for the volumeClaimTemplate fo the StatefulSet
+      #enabled: false
+    #annotations: {}
+
+##The filebeat component can be customized by modifying helm chart values here.
+filebeat:
+  description: "HPCC Managed filebeat"
+  ##See https://github.com/elastic/helm-charts/blob/master/filebeat/values.yaml for all available options
+  labels: {"managedby" : "HPCC"}
+  ## Allows you to add any config files in /usr/share/filebeat
+  ## such as filebeat.yml
+  #filebeatConfig:
+    #filebeat.yml: |
+      #filebeat.inputs:
+      #- type: container
+      #  paths:
+      #    - /var/log/containers/*.log
+        #processors:
+        #- add_kubernetes_metadata:
+            #host: ${NODE_NAME}
+            #matchers:
+            #- logs_path:
+                #logs_path: "/var/log/containers/"
+      #output.elasticsearch:
+      #  host: '${NODE_NAME}'
+      #  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
+      #  index: "hpcc-%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
+
+##The kibana component can be customized by modifying helm chart values here.
+kibana:
+  enabled: true
+  description: "HPCC Managed Kibana"
+  ##See https://github.com/elastic/helm-charts/blob/master/kibana/values.yaml for all available options
+  labels: {"managedby" : "HPCC"}
+  ## Allows you to add any config files in /usr/share/kibana/config/
+  ## such as kibana.yml
+  #kibanaConfig: {}
+  #   kibana.yml: |
+  #     key:
+  #       nestedkey: value