JP5517263B2 - Chunk generating device, chunk reading device, chunk generating method and program - Google Patents

Description

  The present invention relates to a server configuration method for processing information in a sensor information database.

  2. Description of the Related Art Conventionally, as an information management system for handling information such as sensor measurement data, that is, sensor information, there has been uTupleSpace (see Non-Patent Document 1, for example). One of the features of uTupleSpace is that sensor information can be expressed more freely in the “Tuple format” in which “key = value” is arranged. One of the features of uTupleSpace is the introduction of a method for generating chunks of sensor data as means for efficiently storing and retrieving and transferring sensor information.

  Conventionally, there has been a multi-dimensional search tree Ubi-tree (for example, see Non-Patent Document 2) as a data structure for collectively indexing information having a plurality of values as in the u Tuple format.

"Design and implementation of a new uTupleSpace that enables storage and retrieval of large amounts of schemaless data", Keiichiro Kashiwagi et al., Multimedia, Distributed, Collaboration and Mobile (DICOMO2010) Symposium, July 2010 "Improvement of indexing technology UBI-tree for ubiquitous data", Yutaka Arakawa et al., IEICE Technical Report, IEICE Technical Report, vol. 110, no. 162, DE2010-22, pp. 47-52, August 2010

  The simplest method for generating a chunk is to use a certain number of the accumulated sensor information from the oldest in chronological order, starting from the first sensor information accumulation area called the primary DB (or temporary DB). It is a method of moving to another file.

  In order to further improve the efficiency of chunks, considering that search results are handled in units of chunks, data that is often combined as a result with a single search expression, that is, data with similar values, should be merged into the same chunk as much as possible. It is better to gather. To that end, it is conceivable to store a certain amount of sensor information in the temporary DB, extract data having similar values by some means, and transfer them to another file as chunks. .

  At that time, as a method of extracting data having similar values, a method of using a feature of the tree structure and selecting a node arranged nearby in the tree structure is conceivable. At this time, a method of adopting Ubi-tree as a tree structure for managing sensor information is conceivable.

  However, the chunk generation method recalled from the above prior art has a problem that the size of the temporary DB must be kept large. That is, for example, 1 million pieces of sensor information are collected in one chunk file. If similar data is extracted from the population and then combined into one file, the population must be large enough for the previous file to be combined. If this is to be selected from 1000 times the data amount, it means that 1 million × 1000 = 1 billion sensor information must be stored and managed in a single temporary DB.

  Managing such a large size of data increases the load on the memory and the secondary storage device, and the processing load of extracting similar data from such a large amount of data. There was a problem that it would be enormous.

  Conventionally, as disclosed in Non-Patent Document 1, “chunks are stored in a text file format in which data rows are arranged”. In this way, if the chunk is in a text file format in which sensor information is enumerated line by line, the application program can easily create a process to read the chunk file, and the contents of the chunk file are also human readable. There was an advantage that it was easy to handle. On the other hand, in order to obtain only the data rows that are really necessary from the chunk file as a search result, the entire chunk file must be read and a match determination process must be performed for all the rows. There was a problem of becoming higher.

  Alternatively, as a chunk file format, in addition to the description of sensor information in text, a method of separately creating index information such as a tree structure for speeding up the search and storing it as the same file or a different file is conceivable. According to this method, only a part of the chunk file needs to be read in order to obtain only the data rows that are really necessary from the chunk file, and it is necessary to perform a match determination process for all the rows. There is an advantage that there is no. On the other hand, it is not easy for the application program to create a process for reading a chunk file, and the contents of the file are unreadable for humans and difficult to handle.

  That is, there is a problem that the ease of chunk processing and the speed of processing cannot be compatible.

  A chunk generation device according to the present invention is a device that generates a chunk that is a file listing a plurality of information, wherein the device holds a temporary pool, a first chunk, and a chunk pool, and the temporary pool stores information in a tree. The first chunk is a file that enumerates a smaller number of information than the enumerated number of information that should be included in the chunk to be generated by the device, and the chunk pool is data of each of the first chunks. The tag information indicating the range and the identification information of the first chunk is managed in a tree structure, the information is registered in the temporary pool, and the neighboring information group is extracted from the temporary pool in a tree structure to create the first chunk. A function for generating tag information of the first chunk and registering it in the chunk pool, and a tag information in the vicinity of the chunk pool in a tree structure. It has a function of creating a second chunk taken out group includes information including the first chunk showing their tag information at least, the least.

  When the second chunk is created from information included in the k first chunks, the second chunk searches the position in the file of the contents of the k first chunks when creating the second chunk. It may include at least the index information and the arrangement of the k first chunk contents as its constituent elements.

  The temporary pool and the chunk pool use the UBI-Tree search tree algorithm as the tree structure, and the data range of the first chunk indicated by the tag information of the first chunk is expressed as a set of values for a plurality of keys. The index information of the second chunk may be index information for searching the file position of the contents of the first chunk from a set of values for a plurality of keys.

  A chunk reading device according to the present invention is a device that reads a chunk, receives a chunk created by the chunk generation device as an input, reads index information, selects index information that matches a search condition, and a file It has at least a function of moving the reading position to a position in the file indicated by the index information and a function of reading the first chunk contents from the moved position in the file.

The chunk generation method according to the present invention includes a temporary pool, a first chunk, and a chunk pool, and includes a temporary pool registration unit, a first chunk creation unit, a tag information generation unit, and a second chunk creation unit, and includes a plurality of pieces of information. a method chunk generating apparatus for generating a chunk is listed file is executed, the temporary pool registration unit registers the information on the one o'clock pool, the temporary pool, manages the information in a tree structure procedure When the first chunk creation unit, the one o'clock retrieve information group in the vicinity of a tree structure from the pool, the first chunk that lists the number of information less than enumerating the number of generation try to information should contain chunk a step of creating, the tag information generating unit generates the tag information indicating identification information of the first chunk is registered in the chunk pool, the chunk pool And procedures for managing tag information of each of the said first chunk of data range first chunk in a tree structure, the second chunk creation unit extracts the tag information group in the vicinity of a tree structure from the chunk pool, they a step of creating at least comprises second chunk information including the first chunk indicated tag information, the order to execute.

Chunk generating method according to the present invention, the second chunk creation unit, in preparing the second chunk from information including the k-number of the first chunk, the k-number of first chunk in the second chunk at least including Umate contents of the index information for searching a file in position and the arrangement of the k-number of first chunk contents as its components may be.

  The temporary pool and the chunk pool use the UBI-Tree search tree algorithm as the tree structure, and the data range of the first chunk indicated by the tag information of the first chunk is expressed as a set of values for a plurality of keys. The index information of the second chunk may be index information for searching the file position of the contents of the first chunk from a set of values for a plurality of keys.

The chunk generation program according to the present invention causes a computer to hold a temporary pool, a first chunk, and a chunk pool. In the temporary pool, information is managed in a tree structure, and the first chunk includes a chunk to be generated. It is a file that enumerates a smaller number of information than the number of information to be included, and in the chunk pool, tag data indicating the data range of each first chunk and the identification information of the first chunk is managed in a tree structure And a program for realizing each function of the chunk generation device according to the present invention.

  As described above, according to the present invention, chunk generation can be efficiently performed, and the generated chunk can be read easily and at high speed. A chunk generation program can be realized.

The apparatus structure in this Embodiment is shown. The data structure which a temporary pool (101) hold | maintains is shown. The contents of each sensor information (102) stored in the temporary pool (101) are shown. The contents of the sensor information (102) to be newly registered in the temporary pool (101) are shown. The data structure which a temporary pool (101) after performing a registration process hold | maintains is shown. The operation of extracting a nearby sensor information group from the temporary pool (101) is shown. The contents of the created file of the S chunk (104) with the file name S7 are shown. The data structure which S chunk pool (105) hold | maintains is shown. Of the tag information (106) held by the S chunk pool (105), the contents of the file of the S chunk (104) corresponding to S1 and S2 are shown. Indicates the contents of new tag information (301) to be registered in the S chunk pool (105) as the neighboring sensor information group is extracted from the temporary pool (101) and the S chunk (104) is generated. . The data structure which S chunk pool (105) after performing a registration process shows is shown. The operation of extracting a nearby tag information group from the S chunk pool (105) is shown. The contents of the created file of the L chunk (108) having the file name L03 are shown.

One embodiment of the present invention will be described below.
FIG. 1 shows an apparatus configuration in the present embodiment.
The chunk generation device (100) is connected to the network (109) and includes a temporary pool registration unit (110), an S chunk creation unit (111), an S chunk pool registration unit (112), and an L chunk creation unit (113). To do. The S chunk corresponds to the first chunk and the L chunk corresponds to the second chunk.

  Furthermore, the chunk generation device (100) includes a temporary pool (101) for holding data and an S chunk pool (105). The temporary pool (101) holds sensor information (102) with a binary tree data structure, and the S chunk pool (105) holds tag information (106) with a binary tree data structure. The S chunk pool (105) corresponds to the chunk pool.

  Furthermore, the chunk generation device (100) includes an S chunk storage device (103) that holds an S chunk and an L chunk storage device (107) that holds an L chunk.

FIG. 2 shows the data structure held by the temporary pool (101).
In the example of this figure, six pieces of sensor information are held. For example, the description “35: (A)” indicates that the node in the tree holds the information body of (A) shown in FIG. 3 and uses the specific value “35” included in the information body as a main key. Indicates that it is managed by the structure.

  Because of the known characteristics of the binary tree structure, these pieces of sensor information are stored in an aligned manner. That is, when the vertical axis is ignored and only the left and right axes are focused, the main keys are aligned in the order of 0 → 35 → 333 → 599 → 2017 → 3776 from the left in the example of this figure. Such a binary tree data structure management (insertion / retrieval / deletion) method is widely known.

  It is to be noted that the fact that close data is arranged and stored so as to be arranged in the vicinity as in this example is not a feature that is seen only in the binary tree but a feature that is widely observed in the tree structure. Because, in the first place, because the tree structure needs to be searched at high speed, there is a fundamental reason that data is indexed by ordering, so that the stored data is ordered and data close to the neighborhood are arranged. This is a fundamental request in principle.

FIG. 3 shows the contents of each sensor information (102) held in the temporary pool (101).
The sensor information in the present embodiment is information such that the data size of each piece of information is relatively small and includes a plurality of values therein. The present invention functions effectively for sensor information having such characteristics. In particular, in the present embodiment, each sensor information is described in a uTuple data format composed of an arbitrary number of “key = value” sequences.

  The information referred to in the present invention is a variety of information satisfying the above characteristics, and is not limited to the sensor information (102). Specific examples include, but are not limited to, values measured by sensor devices including temperature, humidity, current or voltage values, fluid flow rates, substance concentrations, brightness, noise, position, and acceleration. Information other than sensors, for example, acquired via the Web or the Internet may be used. Further, in addition to these values, information including metadata indicating sensor characteristics and states, measurement date and time, and the like may be used.

  In the example of this figure, each of the three keys A, D, and T has a numeric value, A is an altitude, D is a date, and T is a temperature. . For example, the sensor information in (a) indicates that the altitude is 35 m, the date is June 11, 2011, the temperature is 23.5 degrees Celsius, and the temperature is measured at the highest mountain in Tokyo's 23 wards. Is shown. Similarly, the sensor information in (c) is the temperature data measured at the highest mountain in Tokyo (altitude 2017m), and the sensor information in (d) is the temperature data measured at the highest mountain in Japan (altitude 3776m). Show. In this embodiment, the temporary pool (101) is managed by applying the binary tree data structure to the key “A” (altitude). Note that the scope of application of the present invention is not limited to sensor information described in the u Tuple data format, and is applicable to sensor information having the above-described features in general.

FIG. 4 shows the contents of sensor information (102) to be newly registered in the temporary pool (101).
In the present embodiment, a state in which the sensor information (K) shown in the figure newly arrives at the temporary pool registration unit (101) through the network (109) and performs registration processing of the information will be described in detail below. To do.

  The temporary pool registration unit (110) receives the information and inserts the information into the temporary pool (101) to perform registration processing of the sensor information (102) in the temporary pool (101). The value is 45 because the primary key of the information is A (altitude). The method of inserting this into a binary tree structure is well known, and the result is as shown in FIG.

FIG. 5 shows the data structure held by the temporary pool (101) after the registration process.
Here, the chunk generation device (100) starts the following series of operations for creating an S chunk from the temporary pool. The activation trigger may be the completion of the registration process of the sensor information (102) in the temporary pool (101) by the temporary pool registration unit (110) described above, or asynchronous with the completion of the registration process. In particular, the operation may be activated by means such as a timer.

  First, the activated S chunk creation unit (111) extracts a nearby sensor information group from the temporary pool (101). Here, the neighborhood means that the data values are close to each other, and it can also be said that the data are arranged adjacent to each other on the tree structure based on the above-described consideration regarding the characteristics generally found in the tree structure. Specifically, the processing is as follows.

FIG. 6 shows an operation of extracting a nearby sensor information group from the temporary pool (101).
The S chunk creation unit (111) selects a specific node as a point of interest (261), selects data in the vicinity thereof, and defines it as an extraction range (262). Here, it is assumed that the number of data included in the extraction range is 3 with respect to the number of data in the tree structure, and that (a) is selected as the point of interest (261), and the data of the point of interest ( A) and data (e) (ki) constituting the subtree are taken out as the extraction range.

  In this example, the method of selecting the attention point (261) selects the data with the oldest date. In addition, when selecting data with the newest date, selecting data with the largest or smallest value for a key other than the date (for example, a key other than D, ie, A or T in the example of FIG. 3), It is possible to select the data extracted most frequently by search, or select a candidate having the smallest value width of the entire data included in the extraction range by selecting a plurality or all of the points of interest. .

  The S chunk creation unit (111) reads the sensor information (102) of the extraction range (262) determined as described above from the temporary pool (101), and writes the contents in a new file. This file corresponds to the S chunk (104). The file is stored in the S chunk storage device (103), and a file name that does not overlap with the existing S chunk (104) is assigned. In this example, a serial number is assigned and the file is created with the file name “S7”. Further, the extracted sensor information (102) is deleted from the temporary pool (101).

FIG. 7 shows the contents of the created file of the S chunk (104) having the file name S7.
Descriptions of sensor information (e) (a) (g) included in the extraction range (262) in the utuple data format are written in a text format in the order of the value of the primary key ("A" in this embodiment). Yes.

  Next, with the creation of the new S chunk (104) described above, the S chunk pool registration unit (112) newly updates the tag information (106) corresponding to the S chunk to the S chunk pool (105). Process to register with. In showing the processing contents, first, after showing the state of data held before the registration processing, the procedure of the registration processing will be specifically described.

FIG. 8 shows a data structure held by the S chunk pool (105).
In the example of this figure, six tag information is held. For example, the description “50 to 85: S1” is an example of the tag information, indicating that the node of the tree is tag information corresponding to the information body of the file name S1 shown in FIG. As described above, the tag information does not include the information body, and is referred to as a pointer to the corresponding information body (here, the file name “S1”) and the value of the primary key used for the search (here, “50 to 85”). Contains information only.

  When storing such range information as a primary key in a tree structure, if the tree structure used by the S chunk pool (105) is a data structure that can be directly manipulated using a value range or a plurality of values as an index, the above example If so, it may be stored in the tree structure using two numerical values of 50 and 85. Alternatively, in the present embodiment, a binary tree is used, and only a single value can be directly operated as an index. Therefore, a tree structure is configured using only the opening value “50”. That is, the six nodes in FIG. 8 are arranged in the order of 45 → 50 → 65 → 85 → 333 → 599 in order from the left when paying attention to the opening price. Regardless of which tree structure is used, as described above, close data is arranged and stored so as to be arranged in the vicinity as in this example.

FIG. 9 shows the contents of the file of the S chunk (104) corresponding to S1 and S2 in the tag information (106) held by the S chunk pool (105).
For example, regarding the S chunk (104) having the file name S1, the minimum value in the file of the value of the key “A” is 50 and the maximum value is 85. Therefore, the tag information (106) corresponding to the S chunk has the above value range “50 to 85” and the file name “S1” as values. Similarly, for the S chunk (104) with the file name S2, the corresponding tag information (106) has the value range “45 to 65” and the file name “S2” as values.

  The state of the data held before the registration processing of the tag information (106) by the S chunk pool registration unit (112) has been described above, and the procedure of the corresponding recording processing will be described below.

  FIG. 10 shows the new tag information (301) to be registered in the S chunk pool (105) as the sensor information group in the vicinity is extracted from the temporary pool (101) and the S chunk (104) is generated. The contents of

  In the S chunk (104) having the file name S7 created by the above procedure, the minimum value of the value of the key “A” is 0 and the maximum value is 45 as shown in FIG. Therefore, the S chunk pool registration unit (112) generates the tag information (301) of S07 having the value range “0 to 45” and the file name “S7” as values corresponding to the S chunk.

  Next, the S chunk pool registration unit (112) inserts the generated S7 tag information (301) into the S chunk pool (105), thereby transferring the tag information (106) to the S chunk pool (105). Registration process. In the present embodiment, the opening price “0” of the tag information is inserted into the binary tree structure as the main key. The insertion method is well known, and the result is as shown in FIG.

FIG. 11 shows the data structure held by the S chunk pool (105) after the registration process.
Here, the chunk generation device (100) starts the following series of operations for creating an L chunk from the S chunk pool. The activation trigger may be the completion of the registration process of the tag information (106) to the S chunk pool (105) by the S chunk pool registration unit (112) described above, or the completion of the registration process. May be activated asynchronously by means such as a timer.

  First, the activated L chunk creation unit (113) takes out a nearby tag information group from the S chunk pool (105). Here, the neighborhood means that the data values are close to each other, and it can also be said that the data are arranged adjacent to each other on the tree structure based on the above-described consideration regarding the characteristics generally found in the tree structure. Specifically, the processing is as follows.

FIG. 12 shows an operation of extracting a nearby tag information group from the S chunk pool (105).
The L chunk creation unit (113) selects a specific node as an attention point (321), selects data in the vicinity thereof, and defines it as an extraction range (322). Here, for the number of data in the tree structure, the number of data included in the extraction range operates with a constant of three, and the point of interest (321) and the data constituting the subtree are set as the extraction range.

  In this example, the method of selecting the attention point (321) selects the data having the oldest S chunk generation date, that is, the data having the smallest serial number of the S chunk file name. In addition, select the data with the newest generation date, select the individual row stored in the S chunk, that is, select the data with the largest or smallest value for any key of the individual sensor information, or search the most frequently It is possible to select the data that has been extracted by the above method, or select a candidate having the smallest value width of the entire data included in the extraction range by selecting a plurality or all of the attention points.

  The L chunk creation unit (113) reads the tag information (106) in the retrieval range (322) determined as described above from the S chunk pool (105), and reads the S chunk (106) corresponding to the read tag information (106). 104) is read from the S chunk storage device (103), and the contents are written in a new file in the file format shown in FIG. The file corresponds to the L chunk (108). The file is stored in the L chunk storage device (107), and a file name that does not overlap with the existing L chunk (108) is assigned. In this example, a serial number is assigned and the file is created with the file name “L03”. Further, the retrieved tag information (106) is deleted from the S chunk pool (105).

  The chunk generation device (100) converts the file of the L chunk (108) stored in the L chunk storage device (107) into a file such as NFS, CIFS, FTP, HTTP, WebDAV, GoogleFS, or Hadoop DFS. You may make it public to another computer via a network (109) using a sharing means.

FIG. 13 shows the contents of the created file of the L chunk (108) having the file name L03.
The file is composed of four blocks, and the second to fourth blocks are a copy of the file contents of the S chunk (104) read in the above procedure. In this example, the extraction range (322) indicates tag information corresponding to the three S chunks S7, S2, and S1, so the file contents of these S chunks are copied in order, and the delimiter “[EOB] "At the end. These blocks store information bodies of sensor information.

  Further, the head block of the file is a tag in which tag information of the subsequent block is arranged by the number of lines of the number of the subsequent block, and a delimiter is added at the end. Here, the tag information means information that does not include the information body but includes only information such as a pointer to the corresponding information body and the value of the primary key used for the search. For example, in the example of the first line, the hexadecimal notation of the offset information indicating the number of bytes starting from the beginning of the file in which the first S chunk body is stored as a pointer to the information body (that is, the second block). And information indicating that the value range is “0 to 45” for the key “A”.

  As described above, since the tag information is described in the first block of the L chunk (108), the process of reading the file of the L chunk (108) and searching for necessary sensor information therein is efficiently performed as follows. Can be realized. The operation of the chunk reading apparatus according to the present invention will be described below.

The chunk reading device connected to the network (109) transfers the L chunk file generated in the chunk generation device (100) using the file sharing means.
Next, the contents of the file are read and searched according to the following procedure.

For the sake of explanation, for example, it is assumed that the chunk reading device has transferred the file “L03” of the L chunk, and wants to investigate the temperature measured in an altitude range of 10 m to 40 m.
First, the apparatus reads only the first block of the file into the memory.
Next, for the tag information described in each row of the first block, the device extracts a row in which the overlapping set of the primary key value range and the survey range is not an empty set. In this example, the information in the range of “A = 0 to 45” and the survey range of “10 to 40” have a non-empty range in which the overlapping set is 10 to 40, so this row is extracted. Other rows are not extracted because the overlap is an empty set.

Next, the device specifies the position of the information body according to the pointer described in the tag information of the extracted row.
In this example, the hexadecimal value following the portion “_offset =” on the first line is offset information from the beginning in the file for the second block in which the corresponding information body is stored.

  The device then reads the information body from the location onto the memory. That is, the file is sought, and the file is read from that position until a delimiter appears.

Next, the apparatus examines each row of the read information body on the memory, and extracts a row that matches the examination range.
In this example, the sensor information for three rows is described in the second block, and among them, it is “A = 35, D = 201110611, T = 23. Since this is only the line “5”, this line is extracted. Since this is a search result, the conclusion is that the temperature to be investigated is “23.5 degrees Celsius”.

  In the above-described series of procedures, although the sensor information of 9 pieces is stored in the L chunk, the number of rows in which the sensor information body is actually read from the file into the memory is 3. Even including the tag information, it was 3 + 3 = 6 lines. If a similar search is performed by reading a file in which 9 pieces of sensor information are written in a flat form, it is necessary to read 9 lines of information from the file into the memory. It can be said that the above processing has realized efficient reading and searching processing.

  In the present embodiment, an example in which each S chunk includes 3 pieces of sensor information and each L chunk includes 3 × 3 = 9 pieces of sensor information has been described. However, when this is set to 100 × 100 pieces or 1000 × 1000 pieces, The efficiency difference is further increased, and the superiority of the processing in the present embodiment is further increased. This efficiency is due to the fact that the file format of the L chunk in this embodiment forms a tree structure that is a single type. That is, the L chunk forms a two-layer tree structure in which the first block is a node (root node) in the first layer of the tree and the second to last blocks are nodes in the second layer of the tree.

  In the example of the present embodiment, the root node has three subsequent blocks, and each subsequent block includes sensor information for three rows, thus forming a tri-tree. Since it is a tree structure in this way, it is possible to efficiently search for sensor information in the L chunk by using the above-described chunk reading device using the characteristics of the tree structure.

  Further, the file format of the L chunk is described in a text format while forming a tree structure, and there is an advantage that it is not necessary to include difficult-to-read information in a binary format. Specifically, since the second and subsequent blocks are arranged in the uTuple data format of sensor information as they are, they are highly readable for humans, and a program for reading the entire sensor information can be created very easily. .

  Furthermore, since the first block has a simple format, it is highly readable for humans, and a program for reading or skipping tag information can be easily created. One of the characteristics of the file format of the L chunk of the present embodiment is that both such ease of handling and efficiency due to the tree structure are compatible.

  If this embodiment is used, the file format of the L chunk that enables such an efficient reading process can be efficiently generated using a two-stage tree structure. Such a method of the present embodiment uses the conventional method, that is, one-stage tree structure, that is, only the temporary pool, extracts nine pieces of sensor information, and further rearranges the nine pieces of sensor information to perform the above-described processing of the L chunk. Compared to the method of generating the file structure, the rearrangement process is not necessary, and the L chunk can be generated efficiently.

  Furthermore, in the method of the present embodiment, the size of the temporary pool can be kept small compared to the conventional method using only the one-stage tree structure. This is because the number of populations must be sufficiently larger than the number to be extracted in order to effectively collect sensor information in the vicinity and set it as the extraction range. For example, when the ratio is 0.1%, that is, 1000 times the number is necessary, when storing 1000 × 1000 = 1 million pieces of sensor information in each L chunk, the conventional method has 1000 × 1000 × 1000 = 10. The temporary pool must hold 100 million pieces of sensor information, and the processing load is high.

  According to this embodiment, since 1000 pieces are taken out from the temporary pool, it is sufficient to hold 1000 × 1000 = 1 million pieces, and since the S chunk pool is also taken out 1000 pieces, 1000 × 1000 = 1 million pieces are held. Therefore, the processing load can be greatly reduced, and the target L chunk can be generated efficiently.

  In the present embodiment, a final target chunk file (here, L chunk) is generated using a two-stage tree structure of a temporary pool and an S chunk pool. The scope of the present invention is not limited to this. The same method as that applied to the S chunk is also applied to the L chunk to form an L chunk pool and collect a plurality of L chunks. Chunk generation by a three-stage tree structure that generates “LL chunk” is also possible. Furthermore, although it is possible to have four or five stages in the same manner, about two to three stages are appropriate because the advantages of the ease of processing described above are diminished.

  In the present embodiment, the temporary pool (101) and the S chunk pool (105) are realized using a binary tree data structure. The scope of the present invention is not limited to this, and an arbitrary tree structure including an AVL tree, a B tree, a B + tree, an R tree, or the like can be applied to one or both of them. In particular, the multidimensional search tree Ubi-tree can be applied, and in that case, the following significant advantages can be obtained. That is, Ubi-tree is a data structure that can be processed using a plurality of values and value ranges as an index at a time.

  Furthermore, the types of keys, that is, the number of dimensions, can be handled on the scale of several hundreds. Furthermore, even if a dimension including a value and a dimension not including each data coexist, processing can be performed efficiently. From these characteristics, the Ubi-tree can directly store the sensor information described in the u Tuple data format and perform the search process. When such a Ubi-tree is used for the temporary pool and the S chunk pool of this embodiment, all values as the primary key, that is, the entire uTuple data can be stored in the tree structure as it is. As a result, a set of sensor information or tag information that most closely forms the neighborhood after taking all values into consideration can be obtained as the extraction range.

  Further, since the range of the primary key is described in the tag information, in the Ubi-tree that can take all values as the primary key, the range of all values can be described in the tag information. As a result, the range of all values is listed in the tag information described in the first block of the L chunk, so in the example shown above, only the range condition of altitude, that is, “A”, could be searched efficiently. If Ubi-tree is used, an efficient search within the L chunk can be made for all types of range conditions or combinations thereof. In this case, it is particularly pointed out that the “certain tree structure” formed by the L chunk pointed out above is naturally a Ubi-tree structure itself.

  As described above, by using Ubi-tree in the tree structure in the present embodiment, a remarkable effect can be obtained by a synergistic effect with the tag information generation method and the L chunk file format.

  The apparatus of the present invention can be realized by a computer and a program, and can be recorded on a recording medium or provided through a network.

  The present invention can be applied to the information communication industry.

100: Chunk generation device 101: Temporary pool 102: Sensor information 103: S chunk storage device 104: S chunk 105: S chunk pool 106: Tag information 107: L chunk storage device 108: L chunk 111: S chunk creation unit 112: S chunk pool registration unit 113: L chunk creation unit 261, 321: attention point 262, 322: extraction range 301: tag information of S7

Claims (8)

A device for generating a chunk, which is a file listing a plurality of information,
The device maintains a temporary pool, a first chunk and a chunk pool;
The temporary pool manages information in a tree structure,
The first chunk is a file that lists a smaller number of information than the number of information that should be included in the chunk that the device intends to generate,
The chunk pool manages tag information indicating a data range of each first chunk and identification information of the first chunk in a tree structure,
A function of registering information in the temporary pool;
A function of extracting a neighboring information group in a tree structure from the temporary pool and creating the first chunk;
A function of generating tag information of the first chunk and registering it in the chunk pool;
A function of extracting a neighboring tag information group in a tree structure from the chunk pool and creating a second chunk including at least information included in the first chunk indicated by the tag information;
A chunk generation device characterized by comprising: The chunk generation device according to claim 1,
The function of creating the second chunk is as follows:
When creating the second chunk from the information contained in k first chunks,
The second chunk includes at least index information for searching the position of the k first chunk contents in a file and the arrangement of the k first chunk contents as a constituent element. . The chunk generation device according to claim 1, wherein:
The temporary pool and the chunk pool use a UBI-Tree search tree algorithm as the tree structure,
The data range of the first chunk indicated by the tag information of the first chunk is expressed as a set of values for a plurality of keys,
The chunk generation device, wherein the index information of the second chunk is index information for searching a file position of the contents of the first chunk from a set of values for a plurality of keys. A device that reads chunks,
A function of reading the index information with the chunk created by the chunk generation device according to claim 2 as input;
A function to select index information that matches the search conditions;
A function to move the file reading position to the position in the file indicated by the index information;
A function of reading the contents of the first chunk from the position in the moved file;
A chunk reading apparatus characterized by comprising: The temporary pool, the first chunk, and the chunk pool are held, the temporary pool registration unit, the first chunk creation unit, the tag information generation unit, and the second chunk creation unit are provided, and a chunk that is a file listing a plurality of information is generated. A method performed by a chunk generator ,
And procedures wherein the temporary pool registration unit registers the information on the one o'clock pool, the temporary pool, to manage the information in a tree structure,
Wherein the first chunk creating unit extracts the information group in the vicinity of a tree structure from the one o'clock pool, to create the first chunk listed fewer information than enumerating the number of information should contain chunk to be generated Procedure and
The label information generating unit, said first and generates tag information indicating identification information of the chunk registered in the chunk pool, the chunk pool, tag of the data range of each of the first chunk first chunk Procedures for managing information in a tree structure ;
The second chunk creation unit takes out a neighboring tag information group in a tree structure from the chunk pool and creates a second chunk including at least information included in the first chunk indicated by the tag information;
Chunk generation method to execute in order. The chunk generation method according to claim 5,
The second chunk creation unit
When creating the second chunk from the information contained in k first chunks,
Chunks, wherein at least that include in the arrangement of the the index information for searching a file position within the k first chunk contents of the first chunk content of the k-number in the second chunk as its components Generation method. The chunk generation method according to claim 5, wherein:
The temporary pool and the chunk pool use a UBI-Tree search tree algorithm as the tree structure,
The data range of the first chunk indicated by the tag information of the first chunk is expressed as a set of values for a plurality of keys,
The chunk generation method, wherein the index information of the second chunk is index information for searching a file position of the contents of the first chunk from a set of values for a plurality of keys. Let the computer hold the temporary pool, the first chunk, and the chunk pool,
In the temporary pool, information is managed in a tree structure,
The first chunk is a file that lists a smaller number of information than the number of information that should be included in the chunk to be generated.
In the chunk pool, the tag range information indicating the data range of each of the first chunks and the identification information of the first chunks is managed in a tree structure,
The chunk generation program for implement | achieving each function of any one of Claim 1 thru | or 3 .

Images