Big-Data-HPC-Platform/docs/HPCCDataHandling/DataHandling.xml

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<book lang="en_US" xml:base="../">
  <bookinfo>
    <title>HPCC Data Handling</title>

    <mediaobject>
      <imageobject>
        <imagedata fileref="images/redswooshWithLogo3.jpg" />
      </imageobject>
    </mediaobject>

    <author>
      <surname>Boca Raton Documentation Team</surname>
    </author>

    <legalnotice>
      <para>We welcome your comments and feedback about this document via
      email to <email>docfeedback@hpccsystems.com</email> Please include
      <emphasis role="bold">Documentation Feedback</emphasis> in the subject
      line and reference the document name, page numbers, and current Version
      Number in the text of the message.</para>

      <para>LexisNexis and the Knowledge Burst logo are registered trademarks
      of Reed Elsevier Properties Inc., used under license. HPCC Systems is a
      registered trademark of LexisNexis Risk Data Management Inc.</para>

      <para>Other products, logos, and services may be trademarks or
      registered trademarks of their respective companies. All names and
      example data used in this manual are fictitious. Any similarity to
      actual persons, living or dead, is purely coincidental.</para>

      <para> </para>
    </legalnotice>

    <xi:include href="common/Version.xml" xpointer="FooterInfo"
                xmlns:xi="http://www.w3.org/2001/XInclude" />

    <xi:include href="common/Version.xml" xpointer="DateVer"
                xmlns:xi="http://www.w3.org/2001/XInclude" />

    <corpname>HPCC Systems</corpname>

    <xi:include href="common/Version.xml" xpointer="Copyright"
                xmlns:xi="http://www.w3.org/2001/XInclude" />

    <mediaobject role="logo">
      <imageobject>
        <imagedata fileref="images/LN_Rightjustified.jpg" />
      </imageobject>
    </mediaobject>
  </bookinfo>

  <chapter id="Data_Handling">
    <title><emphasis>HPCC Data Handling</emphasis></title>

    <sect1 id="Introduction" role="nobrk">
      <title>Introduction</title>

      <para>There are a number of different ways in which data may be
      transferred to, from, or within an HPCC system. For each of these data
      transfers, there are a few key parameters that must be known.</para>

      <sect2 id="Prerequisites-for-most-file-movements">
        <title><emphasis role="bold">Prerequisites for most file
        movements:</emphasis></title>

        <itemizedlist>
          <listitem>
            <para>Logical filename</para>
          </listitem>

          <listitem>
            <para>Physical filename</para>
          </listitem>

          <listitem>
            <para>Record size (fixed)</para>
          </listitem>

          <listitem>
            <para>Source directory</para>
          </listitem>

          <listitem>
            <para>Destination directory</para>
          </listitem>

          <listitem>
            <para>Dali IP address (source and/or destination)</para>
          </listitem>

          <listitem>
            <para>Landing Zone IP address</para>
          </listitem>
        </itemizedlist>

        <para>The above parameters are used for these major data handling
        methods:</para>

        <itemizedlist>
          <listitem>
            <para>Import - Spraying Data from the Landing Zone to Thor</para>
          </listitem>

          <listitem>
            <para>Export - Despraying Data from Thor to Landing Zone</para>
          </listitem>

          <listitem>
            <para>Copy - Replicating Data from Thor to Thor (within same Dali
            File System)</para>
          </listitem>

          <listitem>
            <para>Copying Data from Thor to Thor (between different Dali File
            Systems)</para>
          </listitem>
        </itemizedlist>

        <para></para>
      </sect2>
    </sect1>

    <sect1 id="Data_Handling_Terms">
      <title>Data Handling Terms</title>

      <para>A <emphasis>spray</emphasis> or <emphasis>import</emphasis> is the
      relocation of a data file from one location (such as a Landing Zone) to
      a Data Refinery cluster. The term spray was adopted due to the nature of
      the file movement – the file is partitioned across all nodes within a
      cluster.</para>

      <para>A <emphasis>despray</emphasis> or <emphasis>export</emphasis> is
      the relocation of a data file from a Data Refinery cluster to a single
      machine location (such as a Landing Zone). The term despray was adopted
      due to the nature of the file movement – the file is reassembled from
      its parts on all nodes in the cluster and placed in a single file on the
      destination.</para>

      <para>A <emphasis>copy </emphasis>is the replication of a data file from
      one Data Refinery cluster to another cluster within the same
      environment.</para>

      <para>A <emphasis>Remote copy </emphasis>is the replication of a data
      file from one Data Refinery cluster to another cluster in a different
      environment.</para>

      <para>A <emphasis>Landing Zone</emphasis> (or Drop Zone) is a physical
      storage location defined in your system's environment. There can be one
      or more of these locations defined. A daemon (DaFileSrv) must be running
      on that server to enable file sprays and desprays.</para>
    </sect1>

    <sect1 id="Working_with_a_data_file">
      <title>Working with data files</title>

      <para>Once you start working with your HPCC system, you will want to
      process some real data, this section shows you how to load data to your
      HPCC system.</para>

      <sect2 id="Cautions_and_Warnings">
        <title>Before you begin</title>

        <para>First, you should consider the size of the data and the capacity
        of your system. A typical production HPCC system would have much more
        data capacity than a development system. The size of the files you
        wish to work with is limited by the size of your system.</para>
      </sect2>

      <sect2 id="Uploading_a_file">
        <title>Uploading a file</title>

        <para>For smaller data files, maximum of 2GB, you can use the
        upload/download file utility in ECL Watch.</para>

        <orderedlist>
          <listitem>
            <para>In your browser, go to the <emphasis role="bold">ECL
            Watch</emphasis> URL displayed example,
            http://nnn.nnn.nnn.nnn:8010, where nnn.nnn.nnn.nnn is your ESP
            Server's IP address.</para>

            <para><informaltable colsep="1" frame="all" rowsep="1">
                <?dbfo keep-together="always"?>

                <tgroup cols="2">
                  <colspec colwidth="49.50pt" />

                  <colspec />

                  <tbody>
                    <row>
                      <entry><inlinegraphic
                      fileref="images/caution.png" /></entry>

                      <entry>Your IP address could be different from the ones
                      provided in the example images. Please use the IP
                      address provided by <emphasis
                      role="bold">your</emphasis> installation.</entry>
                    </row>
                  </tbody>
                </tgroup>
              </informaltable></para>
          </listitem>

          <listitem>
            <para>From ECL Watch page, click on the <emphasis
            role="bold">Upload/download File </emphasis> link in the menu on
            the left side.</para>

            <para><figure>
                <title>Upload/download</title>

                <mediaobject>
                  <imageobject>
                    <imagedata fileref="images/LZimg03-1.jpg"
                               vendor="eclwatchSS" />
                  </imageobject>
                </mediaobject>
              </figure></para>

            <para><phrase> </phrase>Once you click on the Upload/download file
            link, it will take you to the dropzones and files page, where you
            can choose to <emphasis role="bold">Browse</emphasis> your machine
            for a file to upload:</para>

            <para><figure>
                <title>Dropzones</title>

                <mediaobject>
                  <imageobject>
                    <imagedata fileref="images/LZimg04.jpg"
                               vendor="eclwatchSS" />
                  </imageobject>
                </mediaobject>
              </figure></para>
          </listitem>

          <listitem>
            <para>Press the <emphasis role="bold">Browse</emphasis> button to
            browse the files on your local machine, select the file to upload
            and then click <emphasis role="bold">Open</emphasis>
            button.</para>

            <para>The file you selected should appear in the <emphasis
            role="bold">Select a file to upload</emphasis> field.</para>
          </listitem>

          <listitem>
            <para>Press on <emphasis role="bold">Upload Now</emphasis> to
            complete the file upload.</para>
          </listitem>
        </orderedlist>
      </sect2>

      <sect2 id="Uploading_files_w_secure_client">
        <title>Uploading files with a Secure Copy Client</title>

        <para>To upload a large file for processing to your virtual machine,
        you will need a tool that supports the secure copy protocol. In this
        section, we discuss using WinSCP. There are other tools available, but
        the steps are similar.</para>

        <para><orderedlist>
            <listitem>
              <para>Open the WinSCP tool, and login to your Landing Zone node
              using the username and password given.</para>

              <para><informaltable colsep="1" rowsep="1">
                  <tgroup cols="2">
                    <colspec colwidth="80pt" />

                    <colspec colwidth="100pt" />

                    <tbody>
                      <row>
                        <entry>Login ID:</entry>

                        <entry>hpccdemo</entry>
                      </row>

                      <row>
                        <entry>Password:</entry>

                        <entry>hpccdemo</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>

            <listitem>
              <para>Once logged in, it should, navigate automatically to the
              landing zone folder. (/var/lib/LexisNexis/mydropzone)</para>
            </listitem>

            <listitem>
              <?dbfo keep-together="always"?>

              <para>Navigate to where your local file is in the left part of
              the window.</para>

              <para><figure>
                  <title>WinSCP</title>

                  <mediaobject>
                    <imageobject>
                      <imagedata fileref="images/LZimg05.jpg" />
                    </imageobject>
                  </mediaobject>
                </figure></para>
            </listitem>

            <listitem>
              <para>Select the data file to send and copy it to the landing
              zone, using drag-and-drop.</para>
            </listitem>
          </orderedlist></para>
      </sect2>
    </sect1>

    <sect1 id="Data_Handling_Methods">
      <title>Data Handling Methods</title>

      <para>There are several ways to spray, despray, or copy data
      files:</para>

      <itemizedlist>
        <listitem>
          <para>The DFU interface in Ecl Watch</para>
        </listitem>

        <listitem>
          <para>The DFU Plus command line utility</para>

          <para>See the <emphasis>Client Tools</emphasis> manual for
          details</para>
        </listitem>

        <listitem>
          <para>Using ECL Code and FileServices library functions.</para>

          <para>See the <emphasis>ECL Language Reference</emphasis> for
          details.</para>
        </listitem>
      </itemizedlist>

      <sect2 id="Data_Handling_Using_ECL-Watch">
        <title>Data Handling Using ECL Watch</title>

        <itemizedlist>
          <listitem>
            <para>Login to ECL Watch for the environment.</para>

            <para>The URL is the IP address where the ESP Server is installed
            plus the port to which the wssmc service is bound. The default
            port is 8010. For example:<programlisting>http://&lt;ESPserverIP&gt;:8010/</programlisting></para>
          </listitem>

          <listitem>
            <?dbfo keep-together="always"?>

            <para>Click on the <emphasis role="bold">Browse Logical
            Files</emphasis> hyperlink below <emphasis role="bold">DFU Files
            </emphasis>in the menu on the left.</para>

            <para>The Logical Files page displays showing all files with
            logical entries in the Dali Server’s Distributed File
            System.</para>

            <para><graphic fileref="images/DHMan-3.jpg"
            vendor="eclwatchSS" /></para>

            <para>From this page, you can despray or copy any file.</para>
          </listitem>
        </itemizedlist>

        <sect3 id="Desprays">
          <title>Desprays</title>

          <itemizedlist>
            <listitem>
              <para>Locate the file to despray in the list of files, then
              select the arrow graphic on the left hand side, then select
              <emphasis role="bold">Despray</emphasis> from the pop-up
              menu.</para>
            </listitem>

            <listitem>
              <para>Check the <emphasis role="bold">Source</emphasis>
              information that is already filled in.</para>
            </listitem>

            <listitem>
              <para>Provide <emphasis role="bold">Destination</emphasis>
              information.</para>

              <para><informaltable colsep="0" frame="none" rowsep="0">
                  <tgroup cols="2">
                    <colspec align="left" colwidth="122.40pt" />

                    <colspec colwidth="333.00pt" />

                    <tbody>
                      <row>
                        <entry><emphasis
                        role="bold">Destination-Machine</emphasis></entry>

                        <entry>Use the drop list to select the machine to
                        despray to. The items in the list are landing zones
                        defined in the system’s confguration.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Destination-IP
                        Address</emphasis></entry>

                        <entry>This is prefilled based upon the selected
                        machine.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Destination-Local
                        Path</emphasis></entry>

                        <entry>Provide the complete file path of the
                        destination including file name and extention.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Destination-<emphasis
                        role="bold">Network Path</emphasis></emphasis></entry>

                        <entry>The complete network path of the destination
                        including file name and extension. (read only)</entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Overwrite</emphasis></entry>

                        <entry>Check this box to overwrite a file with the
                        same name if it exists.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>

            <listitem>
              <?dbfo keep-together="always"?>

              <para>Press the <emphasis role="bold">Submit</emphasis>
              button.</para>

              <para>The <emphasis role="bold">DFU Workunit</emphasis>
              displays.<graphic fileref="images/DHMan-4.jpg"
              vendor="eclwatchSS" /></para>
            </listitem>

            <listitem>
              <?dbfo keep-together="always"?>

              <para>Press the <emphasis role="bold">Refresh</emphasis> button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished</emphasis> or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>

              <para><graphic fileref="images/DHMan-5.jpg"
              vendor="eclwatchSS" />The Progress window shows a green progress
              bar indicating the percentage of completion, as well as other
              information related to the operation.</para>

              <para>If a job fails, information related to the cause of the
              failure also displays.</para>
            </listitem>
          </itemizedlist>
        </sect3>

        <sect3 id="Spray_Fixed">
          <title><emphasis role="bold">Spray Fixed</emphasis></title>

          <itemizedlist>
            <listitem>
              <para>Click on the <emphasis role="bold">Spray Fixed</emphasis>
              hyperlink below <emphasis role="bold">DFU</emphasis> in the menu
              on the left.</para>

              <para>The <emphasis role="bold">Spray Fixed</emphasis> page
              displays.</para>
            </listitem>

            <listitem>
              <para>Fill in <emphasis role="bold">Source
              </emphasis>information (<emphasis role="bold">Machine, IP, File
              Path, and record length)</emphasis> and the <emphasis
              role="bold">Destination</emphasis> information (<emphasis
              role="bold">Group</emphasis>and <emphasis role="bold">Label
              ).</emphasis></para>

              <para><informaltable colsep="0" frame="none" rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="122.40pt" />

                    <colspec colwidth="333.00pt" />

                    <tbody>
                      <row>
                        <entry align="right"><emphasis
                        role="bold">Source:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Machine</emphasis></entry>

                        <entry>Use the drop list to select the machine where
                        the source file is located.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">IP</emphasis></entry>

                        <entry>IP address of machine from which to spray. This
                        is automatically completed when you select the
                        <emphasis role="bold">Source
                        Machine.</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Local
                        Path</emphasis></entry>

                        <entry>The file path of source file to spray.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Record
                        Length</emphasis></entry>

                        <entry>The size of each record.</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Destination:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Group</emphasis></entry>

                        <entry>Select the name of THOR cluster to spray
                        to.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Label</emphasis></entry>

                        <entry>The logical name that you choose for the
                        file.</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Options:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Overwrite</emphasis></entry>

                        <entry>Check this box to overwrite files of the same
                        name.</entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Replicate</emphasis></entry>

                        <entry><para>Check this box to create backup copies of
                        all file parts in the backup directory (by convention
                        on the secondary drive of the node following in the
                        cluster).</para><para><emphasis role="bold">This
                        option is only available on systems where replication
                        has been enabled.</emphasis></para></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Compress</emphasis></entry>

                        <entry>Check this box to compress the files.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit</emphasis>
              button.</para>

              <para>The <emphasis role="bold">DFU Workunit</emphasis>
              displays.</para>
            </listitem>

            <listitem>
              <?dbfo keep-together="always"?>

              <para>Press the <emphasis role="bold">Refresh</emphasis>button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished </emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>

              <para><informaltable colsep="1" frame="all" rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="52.60pt" />

                    <colspec colwidth="384.80pt" />

                    <tbody>
                      <row>
                        <entry><graphic fileref="images/tip.jpg" /></entry>

                        <entry>You can use the Choose File button to look up
                        files on the selected source machine.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>
          </itemizedlist>
        </sect3>

        <sect3 id="Spray_CSV">
          <title><emphasis role="bold">Spray CSV</emphasis></title>

          <itemizedlist>
            <listitem>
              <para>Click on the <emphasis role="bold">Spray CSV</emphasis>
              hyperlink below <emphasis role="bold">DFU </emphasis>in the menu
              on the left.</para>

              <para>The <emphasis role="bold">Spray CSV</emphasis> page
              displays.</para>
            </listitem>

            <listitem>
              <para>Fill in <emphasis role="bold">Source
              </emphasis>information (<emphasis role="bold">Machine, IP, File
              Path, and record information)</emphasis> and the <emphasis
              role="bold">Destination</emphasis> information (<emphasis
              role="bold">Group</emphasis> and <emphasis
              role="bold">Label</emphasis>).</para>

              <para><informaltable colsep="0" frame="none" rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="122.40pt" />

                    <colspec colwidth="333.00pt" />

                    <tbody>
                      <row>
                        <entry align="right"><emphasis
                        role="bold">Source:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Machine</emphasis></entry>

                        <entry>Use the drop list to select the machine where
                        the source file is located.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">IP</emphasis></entry>

                        <entry>IP address of machine from which to spray. This
                        is automatically completed when you select the
                        <emphasis role="bold">Source
                        Machine.</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Local
                        Path</emphasis></entry>

                        <entry>The file path of source file to spray.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Max Record
                        Length</emphasis></entry>

                        <entry>The length of longest record in the
                        file.</entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Separator</emphasis></entry>

                        <entry>The character used as a separator in the source
                        file.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Line
                        Terminator</emphasis></entry>

                        <entry>The character used as a line terminator in the
                        source file.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Quote</emphasis></entry>

                        <entry>The character used as a quote in the source
                        file.</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Destination:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Group</emphasis></entry>

                        <entry>Select the name of THOR cluster to spray
                        to.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Label</emphasis></entry>

                        <entry>The logical name that you choose for the
                        file.</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Options:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Overwrite</emphasis></entry>

                        <entry>Check this box to overwrite files of the same
                        name.</entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Replicate</emphasis></entry>

                        <entry><para>Check this box to create backup copies of
                        all file parts in the backup directory (by convention
                        on the secondary drive of the node following in the
                        cluster).</para><para><emphasis role="bold">This
                        option is only available on systems where replication
                        has been enabled.</emphasis></para></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Compress</emphasis></entry>

                        <entry>Check this box to compress the files.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit</emphasis>
              button.</para>

              <para>The <emphasis role="bold">DFU Workunit</emphasis>
              displays.</para>
            </listitem>

            <listitem>
              <?dbfo keep-together="always"?>

              <para>Press the <emphasis role="bold">Refresh</emphasis> button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished</emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.<informaltable colsep="1" frame="all"
                  rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="52.60pt" />

                    <colspec colwidth="384.80pt" />

                    <tbody>
                      <row>
                        <entry><graphic fileref="images/tip.jpg" /></entry>

                        <entry>You can use the Choose File button to look up
                        files on the selected source machine.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>
          </itemizedlist>

          <beginpage />
        </sect3>

        <sect3 id="Spray_XML">
          <title><emphasis role="bold">Spray XML</emphasis></title>

          <itemizedlist>
            <listitem>
              <para>Click on the <emphasis role="bold">Spray
              XML</emphasis>hyperlink below <emphasis
              role="bold">DFU</emphasis>in the menu on the left.</para>

              <para>The <emphasis role="bold">Spray XML</emphasis> page
              displays.</para>
            </listitem>

            <listitem>
              <para>Fill in <emphasis role="bold">Source
              </emphasis>information (<emphasis role="bold">Machine, IP, File
              Path, and record information)</emphasis>and the <emphasis
              role="bold">Destination</emphasis> information (<emphasis
              role="bold">Group</emphasis> and <emphasis role="bold">Label
              ).</emphasis></para>

              <para><informaltable colsep="0" frame="none" rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="122.40pt" />

                    <colspec colwidth="333.00pt" />

                    <tbody>
                      <row>
                        <entry align="right"><emphasis
                        role="bold">Source:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Machine</emphasis></entry>

                        <entry>Use the drop list to select the machine where
                        the source file is located.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">IP</emphasis></entry>

                        <entry>IP address of machine from which to spray. This
                        is automatically completed when you select the
                        <emphasis role="bold">Source
                        Machine.</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Local
                        Path</emphasis></entry>

                        <entry>The file path of source file to spray.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Format</emphasis></entry>

                        <entry>Select the file format from the drop
                        list.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Max Record
                        Length</emphasis></entry>

                        <entry>The length of longest record in the
                        file.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Row
                        Tag</emphasis></entry>

                        <entry>The record separator tag in the XML
                        file</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Destination:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Group</emphasis></entry>

                        <entry>Select the name of THOR cluster to spray
                        to.</entry>
                      </row>

                      <row>
                        <entry><emphasis role="bold">Label</emphasis></entry>

                        <entry>The logical name that you choose for the
                        file.</entry>
                      </row>

                      <row>
                        <entry align="right"><emphasis
                        role="bold">Options:</emphasis></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Overwrite</emphasis></entry>

                        <entry>Check this box to overwrite files of the same
                        name.</entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Replicate</emphasis></entry>

                        <entry><para>Check this box to create backup copies of
                        all file parts in the backup directory (by convention
                        on the secondary drive of the node following in the
                        cluster).</para><para><emphasis role="bold">This
                        option is only available on systems where replication
                        has been enabled.</emphasis></para></entry>
                      </row>

                      <row>
                        <entry><emphasis
                        role="bold">Compress</emphasis></entry>

                        <entry>Check this box to compress the files.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit
              </emphasis>button.</para>

              <para>The <emphasis role="bold">DFU Workunit</emphasis>
              displays.</para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Refresh </emphasis>button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished </emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>

              <para><informaltable colsep="1" frame="all" rowsep="0">
                  <tgroup cols="2">
                    <colspec colwidth="52.60pt" />

                    <colspec colwidth="384.80pt" />

                    <tbody>
                      <row>
                        <entry><graphic fileref="images/tip.jpg" /></entry>

                        <entry>You can use the Choose File button to look up
                        files on the selected source machine.</entry>
                      </row>
                    </tbody>
                  </tgroup>
                </informaltable></para>
            </listitem>
          </itemizedlist>

          <beginpage />
        </sect3>

        <sect3 id="Copy">
          <title><emphasis role="bold">Copy</emphasis></title>

          <itemizedlist>
            <listitem>
              <para>Locate the file to copy in the list of files, then click
              on the arrow icon, then select <emphasis role="bold">Copy
              </emphasis>from the pop-up menu..</para>
            </listitem>

            <listitem>
              <para>Fill in <emphasis role="bold">Destination</emphasis> and
              <emphasis role="bold">Options </emphasis>information.</para>

              <informaltable colsep="0" frame="none" rowsep="0">
                <tgroup cols="2">
                  <colspec colwidth="122.40pt" />

                  <colspec colwidth="333.00pt" />

                  <tbody>
                    <row>
                      <entry align="right"><emphasis
                      role="bold">Destination:</emphasis></entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Group</emphasis></entry>

                      <entry>Select the name of THOR cluster to copy
                      to.</entry>
                    </row>

                    <row>
                      <entry align="right"><emphasis
                      role="bold">Note</emphasis></entry>

                      <entry>You can only choose from THOR clusters within the
                      current environment.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Logical
                      File</emphasis></entry>

                      <entry>The logical name for the copied file.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">File
                      Mask</emphasis></entry>

                      <entry>Automatically updated based on logical file name
                      entered.</entry>
                    </row>

                    <row>
                      <entry align="right"><emphasis
                      role="bold">Options:</emphasis></entry>
                    </row>

                    <row>
                      <entry><emphasis
                      role="bold">Replicate</emphasis></entry>

                      <entry><para>Check this box to create backup copies of
                      all file parts in the backup directory (by convention on
                      the secondary drive of the node following in the
                      cluster).</para><para><emphasis role="bold">This option
                      is only available on systems where replication has been
                      enabled.</emphasis></para></entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Wrap</emphasis></entry>

                      <entry>Check this box to keep the number of parts the
                      same and wrap if the target cluster is smaller that the
                      original.</entry>
                    </row>

                    <row>
                      <entry><emphasis
                      role="bold">Overwrite</emphasis></entry>

                      <entry>Check this box to overwrite files of the same
                      name.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Compress</emphasis></entry>

                      <entry>Check this box to compress the files.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Retain Superfile
                      Structure</emphasis></entry>

                      <entry>Check this box to retain the superfile
                      structure.</entry>
                    </row>
                  </tbody>
                </tgroup>
              </informaltable>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit
              </emphasis>button.</para>

              <para>The <emphasis role="bold">DFU Workunit
              </emphasis>displays.</para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Refresh </emphasis>button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished </emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>
            </listitem>
          </itemizedlist>

          <beginpage />
        </sect3>

        <sect3 id="Remote_Copy">
          <title><emphasis role="bold">Remote Copy</emphasis></title>

          <para>Remote Copy allows you to copy data from a Thor cluster
          outside your environment to the one in your environment.</para>

          <itemizedlist>
            <listitem>
              <para>Click on the <emphasis role="bold">Remote Copy
              </emphasis>hyperlink below <emphasis role="bold">DFU
              </emphasis>in the menu on the left.</para>

              <para>The <emphasis role="bold">Copy File </emphasis>page
              displays.</para>
            </listitem>

            <listitem>
              <para>Fill in <emphasis role="bold">Source,
              Destination,</emphasis> and <emphasis role="bold">Options
              </emphasis>information.</para>

              <informaltable colsep="0" frame="none" rowsep="0">
                <tgroup cols="2">
                  <colspec colwidth="122.40pt" />

                  <colspec colwidth="333.00pt" />

                  <tbody>
                    <row>
                      <entry align="right"><emphasis
                      role="bold">Source:</emphasis></entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Logical
                      File</emphasis></entry>

                      <entry>The logical file name in the remote
                      environment.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Source
                      Dali</emphasis></entry>

                      <entry>The Dali Server in the remote environment</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Source
                      Username</emphasis></entry>

                      <entry>A valid user in the remote environment</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Source
                      Password</emphasis></entry>

                      <entry>The password for the user in the remote
                      environment</entry>
                    </row>

                    <row>
                      <entry align="right"><emphasis
                      role="bold">Destination:</emphasis></entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Group</emphasis></entry>

                      <entry>Select the name of THOR cluster to copy
                      to.</entry>
                    </row>

                    <row>
                      <entry align="right"><emphasis
                      role="bold">Note</emphasis></entry>

                      <entry>You can only choose from THOR clusters within the
                      current environment.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Logical
                      Name</emphasis></entry>

                      <entry>The logical name for the copied file.</entry>
                    </row>

                    <row>
                      <entry align="right"><emphasis
                      role="bold">Options:</emphasis></entry>
                    </row>

                    <row>
                      <entry><emphasis
                      role="bold">Replicate</emphasis></entry>

                      <entry><para>Check this box to create backup copies of
                      all file parts in the backup directory (by convention on
                      the secondary drive of the node following in the
                      cluster).</para><para><emphasis role="bold">This option
                      is only available on systems where replication has been
                      enabled.</emphasis></para></entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Wrap</emphasis></entry>

                      <entry>Check this box to keep the number of parts the
                      same and wrap if the target cluster is smaller that the
                      original.</entry>
                    </row>

                    <row>
                      <entry><emphasis
                      role="bold">Overwrite</emphasis></entry>

                      <entry>Check this box to overwrite files of the same
                      name.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Compress</emphasis></entry>

                      <entry>Check this box to compress the files.</entry>
                    </row>

                    <row>
                      <entry><emphasis role="bold">Retain Superfile
                      Structure</emphasis></entry>

                      <entry>Check this box to retain the superfile
                      structure.</entry>
                    </row>
                  </tbody>
                </tgroup>
              </informaltable>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit
              </emphasis>button.</para>

              <para>The <emphasis role="bold">DFU Workunit
              </emphasis>displays.</para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Refresh </emphasis>button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished </emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>
            </listitem>
          </itemizedlist>

          <beginpage />
        </sect3>

        <sect3 id="Modifying_a-DFU-Workunit">
          <title><emphasis role="bold">Modifying a DFU
          Workunit</emphasis></title>

          <para>From the DFU Workunit page, you can modify and run any DFU
          Spray, Despray, Copy, or Remote Copy action using the modified
          settings. This allows you to run similar jobs without filling in
          similar details again. It also allows you to correct any errors that
          might have caused a DFU workunit to fail.</para>

          <para>From any DFU Workunit page:</para>

          <itemizedlist>
            <listitem>
              <para>Press the Modify button.</para>

              <para>The page for the original action displays.</para>
            </listitem>

            <listitem>
              <para>Modify the details, as needed.</para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Submit
              </emphasis>button.</para>

              <para>The <emphasis role="bold">DFU Workunit
              </emphasis>displays.</para>
            </listitem>

            <listitem>
              <para>Press the <emphasis role="bold">Refresh </emphasis>button
              periodically until the status of your request indicates it is
              <emphasis role="bold">Finished </emphasis>or click on the
              <emphasis role="bold">View Progress</emphasis> hyperlink to see
              a progress indicator.</para>
            </listitem>
          </itemizedlist>

          <para></para>

          <para></para>
        </sect3>
      </sect2>
    </sect1>
  </chapter>

  <chapter>
    <title><emphasis>HPCC Data Backups</emphasis></title>

    <sect1 id="Introduction2" role="nobrk">
      <title>Introduction</title>

      <para>This section covers critical system data that requires regular
      backup procedures to prevent data loss.</para>

      <para>There are</para>

      <itemizedlist>
        <listitem>
          <para>The System Data Store (Dali data)</para>
        </listitem>

        <listitem>
          <para>Environment Configuration files</para>
        </listitem>

        <listitem>
          <para>Data Refinery (Thor) data files</para>
        </listitem>

        <listitem>
          <para>Rapid Data Delivery Engine (Roxie) data files</para>
        </listitem>

        <listitem>
          <para>Attribute Repositories</para>
        </listitem>

        <listitem>
          <para>Landing Zone files</para>
        </listitem>
      </itemizedlist>
    </sect1>

    <sect1>
      <title>Dali data</title>

      <para>The Dali Server data is typically mirrored to its backup node.
      This location is specified in the environment configuration file using
      the Configuration Manager.</para>

      <para>Since the data is written simultaneously to both nodes, there is
      no need for a manual backup procedure.</para>
    </sect1>

    <sect1>
      <title>Environment Configuration files</title>

      <para>There is only one active environment file, but you may have many
      alternative configurations.</para>

      <para>Configuration manager only works on files in the
      /etc/HPCCSystems/source/ folder. To make a configuration active, it is
      copied to /etc/HPCCSystems/environment.xml on all nodes.</para>

      <para>Configuration Manager automatically creates backup copies in the
      /etc/HPCCSystems/source/backup/ folder.</para>
    </sect1>

    <sect1>
      <title>Thor data files</title>

      <para>Thor clusters are normally configured to automatically replicate
      data to a secondary location known as the mirror location. Usually, this
      is on the second drive of the subsequent node.</para>

      <para>If the data is not found at the primary location (for example, due
      to drive failure or because a node has been swapped out), it looks in
      the mirror directory to read the data. Any writes go to the primary and
      then to the mirror. This provides continual redundancy and a quick means
      to restore a system after a node swap.</para>

      <para>A Thor data backup should be performed on a regularly scheduled
      basis and on-demand after a node swap.</para>

      <sect2>
        <title>Manual backup</title>

        <para>To run a backup manually, follow these steps:</para>

        <orderedlist>
          <listitem>
            <para>Login to the Thor Master node.</para>

            <para>If you don't know which node is your Thor Master node, you
            can look it up using ECL Watch.</para>
          </listitem>

          <listitem>
            <para>Run this command:</para>

            <programlisting>sudo su hpcc
/opt/HPCCSystems/bin/start_backupnode &lt;thor_cluster_name&gt; </programlisting>

            <para>This starts the backup process.</para>

            <para></para>

            <graphic fileref="images/backupnode.jpg" />

            <para>Wait until completion. It will say "backupnode finished" as
            shown above.</para>
          </listitem>

          <listitem>
            <para>Run the XREF utility in ECL Watch to verify that there are
            no orphan files or lost files.</para>
          </listitem>
        </orderedlist>
      </sect2>

      <sect2 role="brk">
        <title>Scheduled backup</title>

        <para>The easiest way to schedule the backup process is to create a
        cron job. Cron is a daemon that serves as a task scheduler.</para>

        <para>Cron tab (short for CRON TABle) is a text file that contains a
        the task list. To edit with the default editor, use the
        command:</para>

        <programlisting>sudo crontab -e</programlisting>

        <para>Here is a sample cron tab entry:</para>

        <para><programlisting>30 23 * * * /opt/HPCCSystems/bin/start_backupnode mythor 
</programlisting>30 represents the minute of the hour.</para>

        <para>23 represents the hour of the day</para>

        <para>The asterisks (*) represent every day, month, and
        weekday.</para>

        <para>mythor is the clustername</para>

        <para>To list the tasks scheduled, use the command:</para>

        <programlisting>sudo crontab -l</programlisting>

        <para></para>
      </sect2>
    </sect1>

    <sect1 id="Roxie-Data-Backup">
      <title>Roxie data files</title>

      <para>Roxie data is protected by three forms of redundancy:</para>

      <itemizedlist mark="bullet">
        <listitem>
          <para>Original Source Data File Retention: When a query is deployed,
          the data is typically copied from a Thor cluster's hard drives.
          Therefore, the Thor data can serve as backup, provided it is not
          removed or altered on Thor. Thor data is typically retained for a
          period of time sufficient to serve as a backup copy.</para>
        </listitem>

        <listitem>
          <para>Peer-Node Redundancy: Each Slave node typically has one or
          more peer nodes within its cluster. Each peer stores a copy of data
          files it will read.</para>
        </listitem>

        <listitem>
          <para>Sibling Cluster Redundancy: Although not required, Roxie
          deployments may run multiple identically-configured Roxie clusters.
          When two clusters are deployed for Production each node has an
          identical twin in terms of data and queries stored on the node in
          the other cluster.</para>
        </listitem>
      </itemizedlist>

      <para>This provides multiple redundant copies of data files.</para>
    </sect1>

    <sect1>
      <title>Attribute Repositories</title>

      <para>Attribute repositories are stored on ECL developer's local hard
      drives. They can contain a significant number of hours of work and
      therefore should be regularly backed up. In addition, we suggest using
      some form of source version control, too.</para>
    </sect1>

    <sect1>
      <title>Landing Zone files</title>

      <para>Landing Zones contain raw data for input. They can also contain
      output files. Depending on the size or complexity of these files, you
      may want to retain copies for redundancy.</para>
    </sect1>
  </chapter>
</book>