|
|
@@ -18,15 +18,17 @@ Omnia can install Kubernetes or Slurm (or both), along with additional drivers,
|
|
|
|
|
|
|
|
|
|
|
|
-## Installing Omnia
|
|
|
-Omnia requires that servers already have an RPM-based Linux OS running on them, and are all connected to the Internet. Currently all Omnia testing is done on [CentOS](https://centos.org). Please see [Preparation to install Omnia](PREINSTALL_OMNIA.md) for instructions on network setup.
|
|
|
+## Deploying clusters using the Omnia Appliance
|
|
|
+The Omnia Appliance will automate the entire cluster deployment process, starting with provisioning the operating system to servers.
|
|
|
|
|
|
-Once servers have functioning OS and networking, you can use Omnia to install and start Slurm and/or Kubernetes. Please see [Install Omnia using CLI](INSTALL_OMNIA.md) for detailed instructions.
|
|
|
+Ensure all the prerequisites listed in [preparation to install Omnia Appliance](PREINSTALL_OMNIA_APPLIANCE.md) are met before installing the Omnia appliance.
|
|
|
+
|
|
|
+For detailed instructions on installing the Omnia appliance, see [Install Omnia Appliance](INSTALL_OMNIA_APPLIANCE.md).
|
|
|
|
|
|
-## Installing the Omnia appliance
|
|
|
-Ensure all the prerequisites listed in the [PREINSTALL_OMNIA_APPLIANCE](PREINSTALL_OMNIA_APPLIANCE.md) are met before installing the Omnia appliance.
|
|
|
+## Installing Omnia to servers with a pre-provisioned OS
|
|
|
+Omnia can be deploy clusters to servers that already have an RPM-based Linux OS running on them, and are all connected to the Internet. Currently all Omnia testing is done on [CentOS](https://centos.org). Please see [Preparation to install Omnia](PREINSTALL_OMNIA.md) for instructions on network setup.
|
|
|
|
|
|
-For detailed instructions on installing the Omnia appliance, see [INSTALL_OMNIA_APPLIANCE](INSTALL_OMNIA_APPLIANCE.md).
|
|
|
+Once servers have functioning OS and networking, you can use Omnia to install and start Slurm and/or Kubernetes. Please see [Install Omnia using CLI](INSTALL_OMNIA.md) for detailed instructions.
|
|
|
|
|
|
# System requirements
|
|
|
Ensure the supported version of all the software are installed as per the following table and other versions than those listed are not supported by Omnia. This is to ensure that there is no impact to the functionality of Omnia.
|
|
|
@@ -78,90 +80,10 @@ Issue: Hosts do not display on the AWX UI.
|
|
|
Resolution:
|
|
|
* Verify if `provisioned_hosts.yml` is present in the `omnia/appliance/roles/inventory/files` folder.
|
|
|
* Verify if hosts are not listed in the `provisioned_hosts.yml` file. If hosts are not listed, then servers are not PXE booted yet.
|
|
|
-* If hosts are listed in the `provisioned_hosts.yml` file, then an IP address has been assigned to them by DHCP. However, hosts are not displyed on the AWX UI as the PXE boot is still in process or is not initiated.
|
|
|
+* If hosts are listed in the `provisioned_hosts.yml` file, then an IP address has been assigned to them by DHCP. However, hosts are not displayed on the AWX UI as the PXE boot is still in process or is not initiated.
|
|
|
* Check for the reachable and unreachable hosts using the `provisioned_report.yml` tool present in the `omnia/appliance/tools` folder. To run provisioned_report.yml, in the omnia/appliance directory, run `playbook -i roles/inventory/files/provisioned_hosts.yml tools/provisioned_report.yml`.
|
|
|
|
|
|
-# Frequently asked questions
|
|
|
-* Why is the error "Wait for AWX UI to be up" displayed when `appliance.yaml` fails?
|
|
|
- Cause:
|
|
|
- 1. When AWX is not accessible even after five minutes of wait time.
|
|
|
- 2. When __isMigrating__ or __isInstalling__ is seen in the failure message.
|
|
|
-
|
|
|
- Resolution:
|
|
|
- Wait for AWX UI to be accessible at http://\<management-station-IP>:8081, and then run the `appliance.yaml` file again, where __management-station-IP__ is the ip address of the management node.
|
|
|
-
|
|
|
-* What are the next steps after the nodes in a Kubernetes cluster reboots?
|
|
|
- Resolution:
|
|
|
- Wait for upto 15 minutes after the Kubernetes cluster reboots. Next, verify status of the cluster using the following services:
|
|
|
- * `kubectl get nodes` on the manager node provides correct k8s cluster status.
|
|
|
- * `kubectl get pods --all-namespaces` on the manager node displays all the pods in the **Running** state.
|
|
|
- * `kubectl cluster-info` on the manager node displays both k8s master and kubeDNS are in the **Running** state.
|
|
|
-
|
|
|
-* What to do when the Kubernetes services are not in the __Running__ state?
|
|
|
- Resolution:
|
|
|
- 1. Run `kubectl get pods --all-namespaces` to verify the pods are in the **Running** state.
|
|
|
- 2. If the pods are not in the **Running** state, delete the pods using the command:`kubectl delete pods <name of pod>`
|
|
|
- 3. Run the corresponding playbook that was used to install Kubernetes: `omnia.yml`, `jupyterhub.yml`, or `kubeflow.yml`.
|
|
|
-
|
|
|
-* What to do when the JupyterHub or Prometheus UI are not accessible?
|
|
|
- Resolution:
|
|
|
- Run the command `kubectl get pods --namespace default` to ensure **nfs-client** pod and all prometheus server pods are in the **Running** state.
|
|
|
-
|
|
|
-* While configuring the Cobbler, why does the `appliance.yml` fail with an error during the Run import command?
|
|
|
- Cause:
|
|
|
- * When the mounted .iso file is corrupt.
|
|
|
-
|
|
|
- Resolution:
|
|
|
- 1. Go to __var__->__log__->__cobbler__->__cobbler.log__ to view the error.
|
|
|
- 2. If the error message is **repo verification failed** then it signifies that the .iso file is not mounted properly.
|
|
|
- 3. Verify if the downloaded .iso file is valid and correct.
|
|
|
- 4. Delete the Cobbler container using `docker rm -f cobbler` and rerun `appliance.yml`.
|
|
|
-
|
|
|
-* Why does the PXE boot fail with tftp timeout or service timeout errors?
|
|
|
- Cause:
|
|
|
- * When RAID is configured on the server.
|
|
|
- * When more than two servers in the same network have Cobbler services running.
|
|
|
-
|
|
|
- Resolution:
|
|
|
- 1. Create a Non-RAID or virtual disk in the server.
|
|
|
- 2. Check if other systems except for the management node has cobblerd running. If yes, then stop the Cobbler container using the following commands: `docker rm -f cobbler` and `docker image rm -f cobbler`.
|
|
|
-
|
|
|
-* What to do when the Slurm services do not start automatically after the cluster reboots?
|
|
|
- Resolution:
|
|
|
- * Manually restart the slurmd services on the manager node by running the following commands:
|
|
|
- * `systemctl restart slurmdbd`
|
|
|
- * `systemctl restart slurmctld`
|
|
|
- * `systemctl restart prometheus-slurm-exporter`
|
|
|
- * Run `systemctl status slurmd` to manually restart the following service on all the compute nodes.
|
|
|
-
|
|
|
-* What to do when the Slurm services fail?
|
|
|
- Cause: The `slurm.conf` is not configured properly.
|
|
|
- Resolution:
|
|
|
- 1. Run the following commands:
|
|
|
- * `slurmdbd -Dvvv`
|
|
|
- * `slurmctld -Dvvv`
|
|
|
- 2. Verify `/var/lib/log/slurmctld.log` file.
|
|
|
-
|
|
|
-* What to do when when the error "ports are unavailable" is displayed?
|
|
|
- Cause: Slurm database connection fails.
|
|
|
- Resolution:
|
|
|
- 1. Run the following commands:
|
|
|
- * `slurmdbd -Dvvv`
|
|
|
- *`slurmctld -Dvvv`
|
|
|
- 2. Verify the `/var/lib/log/slurmctld.log` file.
|
|
|
- 3. Verify: netstat -antp | grep LISTEN
|
|
|
- 4. If PIDs are in the **Listening** state, kill the processes of that specific port.
|
|
|
- 5. Restart all Slurm services:
|
|
|
- * slurmctl restart slurmctld on manager node
|
|
|
- * systemctl restart slurmdbd on manager node
|
|
|
- * systemctl restart slurmd on compute node
|
|
|
-
|
|
|
-* What to do if Kubernetes Pods are unable to communicate with the servers when the DNS servers are not responding?
|
|
|
- Cause: With the host network which is DNS issue.
|
|
|
- Resolution:
|
|
|
- 1. In your Kubernetes cluster, run `kubeadm reset -f` on the nodes.
|
|
|
- 2. In the management node, edit the `omnia_config.yml` file to change the Kubernetes Pod Network CIDR. Suggested IP range is 192.168.0.0/16 and ensure you provide an IP which is not in use in your host network.
|
|
|
- 3. Execute omnia.yml and skip slurm using __skip_ tag __slurm__.
|
|
|
+# [Frequently asked questions](FAQ.md)
|
|
|
|
|
|
# Limitations
|
|
|
1. Removal of Slurm and Kubernetes component roles are not supported. However, skip tags can be provided at the start of installation to select the component roles.
|
|
|
@@ -183,4 +105,4 @@ It's not just new features and bug fixes that can be contributed to the Omnia pr
|
|
|
* Feedback
|
|
|
* Validation that it works for your particular configuration
|
|
|
|
|
|
-If you would like to contribute, see [CONTRIBUTORS](https://github.com/dellhpc/omnia/b
|
|
|
+If you would like to contribute, see [CONTRIBUTING](https://github.com/dellhpc/omnia/blob/release/CONTRIBUTING.md).
|
Lucas A. Wilson
