HPCC-9593 Added documentation pertaining to cmake structure

Cmake structure outline and brief documentation 1st draft.

Signed-off-by: Philip Schwartz <philip.schwartz@lexisnexis.com>

Reworked a bit prior to pulling.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>

cmake_modules/Find_structure.rst

@@ -0,0 +1,45 @@
+Find*****.cmake
+
+- Will define when done:
+    - *****_FOUND
+    - *****_INCLUDE_DIR
+    - *****_LIBRARIES
+    (more can be defined, but must be at min the previous unless looking for only a binary)
+
+All of our Find scripts use the following format:
+
+NOT *****_FOUND )
+    Externals set
+        define needed vars for finding external based libraries/headers
+
+    Use Native set
+        use FIND_PATH to locate headers
+        use FIND_LIBRARY to find libs
+
+    Include Cmake macros file for package handling
+    define package handling args for find return  (This will set *****_FOUND)
+
+
+    *****_FOUND
+        perform any modifications you feel is needed for the find
+
+    Mark defined variables used in package handling args as advanced for return
+
+
+FindAPR.cmake
+
+IF (NOT APR_FOUND)
+  if (USE_NATIVE_LIBRARIES)
+    # if we didn't find in externals, look in system include path
+    FIND_PATH (APR_INCLUDE_DIR NAMES apr/apr.h)
+    FIND_LIBRARY (APR_LIBRARIES NAMES apr)
+  endif()
+
+  include(FindPackageHandleStandardArgs)
+  find_package_handle_standard_args(APR DEFAULT_MSG
+    APR_LIBRARIES
+    APR_INCLUDE_DIR
+  )
+
+  MARK_AS_ADVANCED(APR_INCLUDE_DIR APR_LIBRARIES)
+ENDIF()

cmake_modules/cmake_common_things.rst

@@ -0,0 +1,35 @@
+Common looping:
+
+FOREACH( oITEMS
+    item1
+    item2
+    item3
+    item4
+    item5
+)
+    Actions on each item here.
+ENDFOREACH ( oITEMS )
+
+Common installs over just install:
+
+    install ( FILES ... ) - installs item with 664 permissions
+    install ( PROGRAMS ... ) - installs runable item with 755 permissions
+    install ( TARGETS ... ) - installs built target with 755 permissions
+    install ( DIRECTORY ... ) - installs directory with 777 permissions
+
+Common settings for generated source files:
+    set_source_files_properties(<file> PROPERTIES GENERATED TRUE) - Must be set on generated source files or dependency generation fails and increases build time.
+
+When using custom commands between multiple cmake files use:
+    GET_TARGET_PROPERTY(<VAR from other cmake file> <var for this file> LOCATION)
+
+    GET_TARGET_PROPERTY(ESDL_EXE esdl LOCATION) - will get from the top level cache the ESDL_EXE value and set it in esdl for your current cmake file
+
+
+USE add_custom_command only when 100% needed.
+
+All directories in a cmake project should have a CMakeLists.txt file and be called to the upper level project with an add_subdirectory or HPCC_ADD_SUBDIRECTORY
+
+When you have a property that will be shared between cmake projects use define_property to set it in the top level cache.
+    define_property(GLOBAL PROPERTY TEST_TARGET BRIEF_DOCS "test doc" FULL_DOCS "Full test doc")
+    mark_as_advanced(TEST_TARGET)  - this is required to force the property into the top level cache.

cmake_modules/cmake_dir_structure.rst

@@ -0,0 +1,116 @@
+CMake Layout:
+
+/
+- CMakeLists.txt - Root CMake file
+- version.cmake - common cmake file where version variables are set
+- build-config.h.cmake - cmake generation template for build-config.h
+
+- cmake_modules/ - Directory storing modules and configurations for CMake
+-- Find*****.cmake - CMake find file used to locate libraries, headers, and binaries
+-- commonSetup.cmake - common configuration settings for the entire project (contains configure time options)
+-- docMacros.cmake - common documentation macros used for generating fop and pdf files
+-- optionDefaults.cmake - contains common variables for the platform build
+-- distrocheck.sh - script that determines if the OS uses DEB or RPM
+-- getpackagerevisionarch.sh - script that returns OS version and arch in format used for packaging
+-- dependencies/ - Directory storing dependency files used for package dependencies
+--- <OS>.cmake - File containing either DEB or RPM dependencies for the given OS
+
+- build-utils/ - Directory for build related utilities
+-- cleanDeb.sh - script that unpacks a deb file and rebuilds with fakeroot to clean up lintain errors/warnings
+
+- initfiles/ - Directory containing init and install based code
+-- CMakeLists.txt - initfiles cmake file which defines GENERATE_BASH macro
+-- bash-vars.in - cmake template file for bash configuration variables that are injected into all installed bash scripts
+-- processor.cpp - simple application used at build time to search and replace ###<REPLACE>### in bash scripts
+
+-- sbin/ - Directory containing administration based scripts
+--- add_conf_settings.sh.in - used to add sudoers and limits.conf settings on package install
+--- alter_confs.sh - contains functions used by add_conf_settings.sh.in and rm_conf_settings.sh.in
+--- complete-uninstall.sh.in - script to remove package and all directories from platform
+--- configmgr.in - configmgr start script
+--- get_ip_address.sh.in - script to get base ip address of server
+--- hpcc-push.sh.in - script to push files to servers defined in environment using ssh keys
+--- hpcc-run.sh.in - script to run init commands on servers defined in environment using ssh keys
+--- hpcc_setenv.in - source-able file that defines hpcc env vars, used by init system
+--- install-cluster.sh.in - script to install platform on a cluster using environment file and expect
+--- keygen.sh.in - script to generate ssh keys for hpcc user
+--- killconfigmgr - script used to kill configmgr when running (used by configmgr start script)
+--- prerm.in - script run pre-remove of the installed DEB or RPM
+--- regex.awk.in.cmake - regex awk code used by configmgr
+--- remote-install-engine.sh.in - payload install script used by install-cluster.sh
+--- rm_conf_settings.sh.in - remove sudoers and limits.conf settings on package uninstall
+
+-- etc/
+--- bash_completion/ - contains bash completion scripts used by the bash shell
+---- ecl - ecl cmd completion
+---- eclagent - eclagent cmd completion
+---- eclcc - eclcc cmd completion
+---- eclplus - eclplus cmd completion
+
+--- DIR_NAME/ - Directory used to generate /etc/<DIR_NAME> on package install (name is important)
+---- environment.conf.in - environment.conf template
+---- environment.xml.in - all in one single node template
+---- genenvrules.conf - environment generation rules used by config wizard
+---- version.in - version file template used by configmgr and esp
+
+---- configmgr/ - Directory containing configmgr based configs
+----- configmgr.conf.in - configmgr config file
+----- esp.xml.in - esp config used to start the configmgr esp process
+
+
+--- sshkey/ - contains base ssh keys included in platform packages
+---- .ssh.md5 - md5 sums of .ssh dir allowing for comparision at platform start/stop for security check
+---- .ssh/ - directory containing key files
+----- authorized_keys - file containing keys for hpcc that can be used for auth
+----- id_rsa - private ssh key
+----- id_rsa.pub - public ssh key
+
+-- bin/ - Directory containing the scripts used to start and stop component processes
+--- init_configesp
+--- init_dafilesrv.in
+--- init_dali
+--- init_dfuserver
+--- init_eclagent.in
+--- init_eclccserver
+--- init_eclscheduler
+--- init_esp
+--- init_ftslave
+--- init_roxie
+--- init_roxie_cluster
+--- init_sasha
+--- init_thor
+
+-- componentfiles/ - Directory containing subdirectories of things used by other components/installed items
+--- configxml - files used by configmgr
+--- ftslave - files used by ftslace
+--- launcher - files used by the unity launcher
+--- thor - files used by thor
+
+-- bash/
+--- etc/
+---- init.d/
+----- dafilesrv.in - dafilesrv init script
+----- export-path
+----- hpcc_common.in - common functions for hpcc scripts
+----- hpcc-init.in - hpcc-init init script
+----- hpcc-init.install -  hpcc-init install script used by package install
+----- hpcc-init.uninstall - hpcc-init uninstall script used by package uninstall
+----- init-functions - common functions related completely to init
+----- install-init.in - script used to install hpcc init system on package install
+----- lock.sh - common functions related to lock files
+----- lsb-base-logging.sh - common functions related to logging to terminal
+----- pid.sh - common functions related to pid files
+----- uninstall-init.in - script used to uninstall hpcc init system on package uninstall
+
+--- sbin/ - Directory containing bash based package install sripts
+---- bash_postinst.in - post install used by RPM package
+---- deb/ - Directory containing DEB specific scripts
+----- postinst.in - post install used by DEB package
+----- postrm.in - post remove used by DEB package
+
+
+- docs/ - Directory for documentation building
+-- bin/ - Directory containing scripts used by documentation team
+-- BuildTools/ - Directory containing xsl files and xsl file templates used by documentation team
+-- common/ - Directory containing common files used by all documents built
+-- resources/ - directory containing docbook resources needed to build documentation

cmake_modules/cmake_doc_process.rst

@@ -0,0 +1,15 @@
+Doc build process:
+
+Configure:
+1. Find xsltproc
+2. Find fop
+3. configure custom xsl and xml files needed for document generation.
+
+Build:
+4. Expand docbook resources
+5. build docs based on calls to DOCBOOK_TO_PDF
+    a. CLEAN_REL_BOOK for provided xml
+    b. RUN_XSLTPROC on xml file generated in a using defined xsl file to generate fo file
+    c. RUN_FOP to generate pdf from fo file generated in b
+
+The entire build process via a call to DOCBOOK_TO_PDF is run on cmake targets that are generated with custom targets at configure time.

cmake_modules/cmake_macros.rst

@@ -0,0 +1,20 @@
+Common Macros for all builds:
+    MACRO_ENSURE_OUT_OF_SOURCE_BUILD - prevents building from with in source tree
+    HPCC_ADD_EXECUTABLE - simple wrapper around add_executable
+    HPCC_ADD_LIBRARY - simple wrapper around add_library
+    PARSE_ARGUMENTS - macro that can be used by other macros and functions for arg list parsing
+    HPCC_ADD_SUBDIRECTORY - argument controlled add subdirectory wrapper
+        HPCC_ADD_SUBDIRECTORY(test t1 t2 t3) - Will add the subdirectory test if t1,t2, or t3 are set to any value other then False/OFF/0
+
+    LOG_PLUGIN - used to log any code based plugin for the platform
+    ADD_PLUGIN - adds a plugin with optional build dependencies to the build if dependencies are found
+
+Document Macros:
+    RUN_XSLTPROC - Runs xsltproc using given args
+    RUN_FOP - Runs fop using given args
+    CLEAN_REL_BOOK - Uses a custom xsl and xsltproc to clean relative book paths from given xml file
+    CLEAN_REL_SET - Uses a custom xsl and xsltproc to clean relative set paths from given xml file
+    DOCBOOK_TO_PDF - Master macro used to generate pdf, uses above macros
+
+Initfiles macro:
+    GENERATE_BASH - used to run processer program on given files to replace ###<REPLACE>### with given variables

cmake_modules/cmake_plugin_structure.rst

@@ -0,0 +1,38 @@
+project( **embed )
+
+if (USE_**)
+  ADD_PLUGIN(**embed PACKAGES ** OPTION MAKE_**EMBED MINVERSION 1.0 MAXVERSION 2.0)
+  if ( MAKE_**EMBED )
+    set ( SRCS
+          **.cpp
+        )
+
+    include_directories (
+         "${**_INCLUDE_DIR}"
+         ./../../system/include
+         ./../../rtl/eclrtl
+         ./../../rtl/include
+         ./../../common/deftype
+         ./../../system/jlib
+       )
+
+    ADD_DEFINITIONS( -D_USRDLL -D**EMBED_EXPORTS )
+
+    HPCC_ADD_LIBRARY( **embed SHARED ${SRCS} )
+    if (${CMAKE_VERSION} VERSION_LESS "2.8.9")
+      message("WARNING: Cannot set NO_SONAME. shlibdeps will give warnings when package is installed")
+    else()
+      set_target_properties( **embed PROPERTIES NO_SONAME 1 )
+    endif()
+
+    install ( TARGETS **embed DESTINATION plugins )
+    target_link_libraries ( **embed ${**_LIBRARY} )
+
+    target_link_libraries ( **embed
+        eclrtl
+        jlib
+        )
+  endif()
+endif()
+
+install ( FILES ${CMAKE_CURRENT_SOURCE_DIR}/**.ecllib DESTINATION plugins COMPONENT Runtime)

cmake_modules/cmake_project_structure.rst

@@ -0,0 +1,20 @@
+project( myproj )
+
+set (    SRCS
+         test.cpp
+    )
+
+include_directories (
+         ${CMAKE_BINARY_DIR}
+         ${CMAKE_BINARY_DIR}/oss
+         ./../../system/include
+         ./../../system/jlib
+    )
+
+ADD_DEFINITIONS( -D_CONSOLE )
+
+HPCC_ADD_EXECUTABLE ( myproj ${SRCS} )
+install ( TARGETS myproj RUNTIME DESTINATION ${EXEC_DIR} )
+target_link_libraries ( myproj
+         jlib
+    )

cmake_modules/hpcc_install_process.rst

@@ -0,0 +1,27 @@
+Init install:
+
+All items are installed to /opt/HPCCSystems by default (can be changed at configure time).
+
+Part of the install copies files to /etc/HPCCSystems. This process will not replace files currently in the directory. It will replace files in /etc/HPCCSystems/rpmnew.
+
+The install-init process makes use of 3 bash functions along with the hpcc_common functions:
+    installConfs
+    installFile
+    fileCheck
+
+installConfs is used to copy config files into place if they do not exist. It will not replace an existing file.
+
+installFile will copy a file from one location to another. This will replace an existing file and also has an option to create as a symlink instead of copying.
+
+fileCheck is used to check for ssh keys for the hpcc user along with warning the user if we are installing or they are still using (upgrade) our publicly provided ssh keys.
+
+Most install procedures are handled in install-init directly, but install-init also supports sub installs using install files that are placed in /opt/HPCCSystems/etc/init.d/install/
+
+(Most files are installed as symlinks)
+
+The final steps of the install are to set permissions correctly for the hpcc user along with calling add_conf_settings.sh to add the sudoers and limits.conf changes.
+
+
+Init uninstall:
+
+This removes all symlinks created during the install process. This also supports sub installs using uninstall files that are placed in /opt/HPCCSystems/etc/init.d/uninstall/