|
@@ -3,11 +3,9 @@
|
|
|
by M. Neteler 8/2005
|
|
|
-->
|
|
|
|
|
|
-\section segmentintro Segment Library
|
|
|
+\author CERL
|
|
|
|
|
|
-
|
|
|
-<P>
|
|
|
-Authors: CERL
|
|
|
+\section segmentintro Introduction
|
|
|
|
|
|
<P>
|
|
|
Large data files which contain data in a matrix format often need to be
|
|
@@ -61,9 +59,8 @@ specifying the row and column of the original matrix. Behind the scenes, the
|
|
|
data is paged into memory as needed and the requested data is returned to
|
|
|
the caller.
|
|
|
|
|
|
-<P>
|
|
|
-<B>Note:</B> All routines and global variables in this library, documented
|
|
|
-or undocumented, start with the prefix <B>segment_.</B> To avoid name
|
|
|
+\note All routines and global variables in this library, documented
|
|
|
+or undocumented, start with the prefix \c segment_. To avoid name
|
|
|
conflicts, programmers should not create variables or routines in their own
|
|
|
modules which use this prefix.
|
|
|
|
|
@@ -74,12 +71,14 @@ modules which use this prefix.
|
|
|
The routines in the <I>Segment Library</I> are described below, more or
|
|
|
less in the order they would logically be used in a module. They use a data
|
|
|
structure called SEGMENT which is defined in the header file
|
|
|
-<grass/segment.h> that must be included in any code using these
|
|
|
-routines: [footnote]
|
|
|
+\c grass/segment.h that must be included in any code using these
|
|
|
+routines:
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
#include <grass/segment.h>
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
+
|
|
|
+\see \ref Loading_the_Segment_Library.
|
|
|
|
|
|
<P>
|
|
|
A temporary file needs to be prepared and a SEGMENT structure needs to
|
|
@@ -131,22 +130,21 @@ Return codes are: 1 ok; else -1 could not seek or write <I>fd</I>, or -3
|
|
|
|
|
|
<P>
|
|
|
The next step is to initialize a SEGMENT structure to be associated with a
|
|
|
-segment file formatted by <I>segment_format.</I>
|
|
|
+segment file formatted by segment_format().
|
|
|
|
|
|
<P>
|
|
|
<I>int segment_init (SEGMENT *seg, int fd, int nsegs)</I>, initialize segment
|
|
|
structure
|
|
|
<P>
|
|
|
Initializes the <B>seg</B> structure. The file on <B>fd</B> is
|
|
|
- a segment file created by <I>segment_format</I> and must be open for
|
|
|
+ a segment file created by segment_format() and must be open for
|
|
|
reading and writing. The segment file configuration parameters <I>nrows,
|
|
|
ncols, srows, scols</I>, and <I>len</I>, as written to the file by
|
|
|
<I>segment_format</I>, are read from the file and stored in the
|
|
|
<B>seg</B> structure. <B>Nsegs</B> specifies the number of segments that
|
|
|
will be retained in memory. The minimum value allowed is 1.
|
|
|
|
|
|
-<P>
|
|
|
-<B>Note.</B> The size of a segment is <I>scols*srows*len</I> plus a few
|
|
|
+\note The size of a segment is <I>scols*srows*len</I> plus a few
|
|
|
bytes for managing each segment.
|
|
|
|
|
|
<P>
|
|
@@ -161,7 +159,7 @@ Then data can be written from another file to the segment file row by row:
|
|
|
<P>
|
|
|
Transfers nonsegmented matrix data, row by row, into a segment
|
|
|
file. <B>Seg</B> is the segment structure that was configured from a call
|
|
|
- to <I>segment_init.</I> <B>Buf</B> should contain <I>ncols*len</I>
|
|
|
+ to segment_init(). <B>Buf</B> should contain <I>ncols*len</I>
|
|
|
bytes of data to be transferred to the segment file. <B>Row</B> specifies
|
|
|
the row from the data matrix being transferred.
|
|
|
|
|
@@ -206,10 +204,10 @@ flushed to disk:
|
|
|
<P>
|
|
|
<I>int segment_flush (SEGMENT *seg)</I>, flush pending updates to disk
|
|
|
<P>
|
|
|
- Forces all pending updates generated by <I>segment_put()</I> to be
|
|
|
+ Forces all pending updates generated by segment_put() to be
|
|
|
written to the segment file <B>seg.</B> Must be called after the final
|
|
|
segment_put() to force all pending updates to disk. Must also be called
|
|
|
- before the first call to <I>segment_get_row.</I>
|
|
|
+ before the first call to segment_get_row().
|
|
|
|
|
|
<P>
|
|
|
Now the data in segment file can be read row by row and transferred to a normal
|
|
@@ -221,7 +219,7 @@ sequential data file:
|
|
|
<P>
|
|
|
Transfers data from a segment file, row by row, into memory
|
|
|
(which can then be written to a regular matrix file) . <B>Seg</B> is the
|
|
|
- segment structure that was configured from a call to <I>segment_init.</I>
|
|
|
+ segment structure that was configured from a call to segment_init().
|
|
|
<B>Buf</B> will be filled with <I>ncols*len</I> bytes of data
|
|
|
corresponding to the <B>row</B> in the data matrix.
|
|
|
|
|
@@ -236,7 +234,7 @@ Finally, memory allocated in the SEGMENT structure is freed:
|
|
|
<P>
|
|
|
Releases the allocated memory associated with the segment file
|
|
|
<B>seg.</B> Does not close the file. Does not flush the data which may
|
|
|
- be pending from previous <I>segment_put()</I> calls.
|
|
|
+ be pending from previous segment_put() calls.
|
|
|
|
|
|
<P>
|
|
|
|
|
@@ -259,81 +257,77 @@ The following should provide the programmer with a good idea of how to use the
|
|
|
Creation of a segment file and initialization of the segment structure
|
|
|
at once:
|
|
|
|
|
|
-\verbatim
|
|
|
-
|
|
|
+\code
|
|
|
SEGMENT seg;
|
|
|
|
|
|
segment_open (&seg, G_tempfile(), nrows, ncols, srows, scols, sizeof(int), nseg);
|
|
|
-
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
Alternatively, the first step is the creation and formatting of a segment
|
|
|
file. A file is created, formatted and then closed:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
fd = creat (file, 0666);
|
|
|
|
|
|
segment_format (fd, nrows, ncols, srows, scols, sizeof(int));
|
|
|
|
|
|
close(fd);
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
<P>
|
|
|
The next step is the conversion of the nonsegmented matrix data into segment
|
|
|
file format. The segment file is reopened for read and write and initialized:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
#include <fcntl.h>
|
|
|
|
|
|
SEGMENT seg;
|
|
|
|
|
|
fd = open (file, O_RDWR);
|
|
|
segment_init (&seg, fd, nseg);
|
|
|
-
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
<P>
|
|
|
Both the segment file and the segment structure are now ready to use, and
|
|
|
data can be read row by row from the original data file and put into the
|
|
|
segment file:
|
|
|
|
|
|
-\verbatim
|
|
|
-
|
|
|
+\code
|
|
|
for (row = 0; row < nrows; row++)
|
|
|
{
|
|
|
- <code to get original matrix data for row into buf>
|
|
|
+ // code to get original matrix data for row into buf
|
|
|
|
|
|
segment_put_row (&seg, buf, row);
|
|
|
}
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
|
|
|
<P>
|
|
|
Of course if the intention is only to add new values rather than update existing
|
|
|
values, the step which transfers data from the original matrix to the segment
|
|
|
-file, using segment_put_row() , could be omitted, since
|
|
|
-<I>segment_format</I> will fill the segment file with zeros.
|
|
|
+file, using segment_put_row(), could be omitted, since
|
|
|
+segment_format() will fill the segment file with zeros.
|
|
|
|
|
|
<P>
|
|
|
-The data can now be accessed directly using <I>segment_get.</I> For example,
|
|
|
+The data can now be accessed directly using segment_get(). For example,
|
|
|
to get the value at a given row and column:
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
int value;
|
|
|
|
|
|
SEGMENT seg;
|
|
|
|
|
|
segment_get (&seg, &value, row, col);
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
<P>
|
|
|
-Similarly <I>segment_put()</I> can be used to change data values in the
|
|
|
+Similarly segment_put() can be used to change data values in the
|
|
|
segment file:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
int value;
|
|
|
|
|
|
SEGMENT seg;
|
|
@@ -341,16 +335,16 @@ SEGMENT seg;
|
|
|
value = 10;
|
|
|
|
|
|
segment_put (&seg, &value, row, col);
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
-<P>
|
|
|
-<B>WARNING:</B> It is an easy mistake to pass a value directly to
|
|
|
+
|
|
|
+\warning It is an easy mistake to pass a value directly to
|
|
|
segment_put(). The following should be avoided:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
-segment_put (&seg, 10, row, col); /* this will not work */
|
|
|
-\endverbatim
|
|
|
+\code
|
|
|
+segment_put (&seg, 10, row, col); // this will not work
|
|
|
+\endcode
|
|
|
|
|
|
<P>
|
|
|
Once the random access processing is complete, the data would be extracted
|
|
@@ -358,30 +352,30 @@ from the segment file and written to a nonsegmented matrix data file as
|
|
|
follows:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
segment_flush (&seg);
|
|
|
|
|
|
for (row = 0; row < nrows; row++)
|
|
|
{
|
|
|
segment_get_row (&seg, buf, row);
|
|
|
|
|
|
- <code to put buf into a matrix data file for row>
|
|
|
+ // code to put buf into a matrix data file for row
|
|
|
}
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
<P>
|
|
|
Finally, the memory allocated for use by the segment routines would be
|
|
|
released and the file closed:
|
|
|
|
|
|
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
segment_release (&seg);
|
|
|
|
|
|
close (fd);
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
|
|
|
-<P>
|
|
|
-<B>Note:</B> The <I>Segment Library</I> does not know the name of the
|
|
|
+
|
|
|
+\note The <I>Segment Library</I> does not know the name of the
|
|
|
segment file. It does not attempt to remove the file. If the file is only
|
|
|
temporary, the programmer should remove the file after closing it.
|
|
|
|
|
@@ -403,9 +397,9 @@ performance, particularly for larger datasets.
|
|
|
|
|
|
<P>
|
|
|
The library is loaded by specifying
|
|
|
-\verbatim
|
|
|
+\code
|
|
|
$(SEGMENTLIB)
|
|
|
-\endverbatim
|
|
|
+\endcode
|
|
|
in the Makefile.
|
|
|
|
|
|
<P>
|