WO2014163072A1

WO2014163072A1 - Information processing apparatus, information processing method, and program

Info

Publication number: WO2014163072A1
Application number: PCT/JP2014/059626
Authority: WO
Inventors: 浩明井上; 竹中　崇
Original assignee: 日本電気株式会社
Priority date: 2013-04-01
Filing date: 2014-04-01
Publication date: 2014-10-09
Also published as: JP6428603B2; JPWO2014163072A1

Abstract

An information processing apparatus is provided with a write control unit, a plurality of data search units, and an output sequence control unit. The write control unit cyclically allots input data included in stream data to the plurality of data search units in sequence. The plurality of data search units stores the allotted input data, and from among the stored input data, searches and outputs the input data that includes the same identifiers as the identifiers included in the retrieval data, and was stored earliest. The output sequence control unit sorts the input data outputted from the plurality of data search units into the sequence of input data of the stream data according to the allotment status from the write control unit, joins the input data outputted from the plurality of data search units with the retrieval data in the post-sort sequence, and outputs the data. Join processing of the stream data is speeded up while retaining the sequence of the data before and after the joining process.

Description

Information processing apparatus, information processing method, and program

[Description of related applications]
The present invention is based on a Japanese patent application: Japanese Patent Application No. 2013-076114 (filed on April 1, 2013), and the entire contents of this application are incorporated in the present specification by reference.
The present invention relates to an information processing device, an information processing method, and a program, and more particularly, to an information processing device, an information processing method, and a program that perform stream data combining processing.

In recent years, by processing data (stream data, time-series data) obtained from information sources such as sensors and terminals in every moment, smarter social infrastructure (for example, diagnosis of deterioration of structures, elimination of traffic congestion, etc.) Information processing technology that aims to achieve this is progressing.

One of the processes required for handling stream data is a process called “join” process. The join process is a search from stream data by searching for data having the same search ID as the search data from a limited range of stream data received in the past using the search ID included in the search data as a key. The data is combined with the search data and output in the order of the data in the stream data received in the past. As a specific example of the combining process, there is a process of combining the price increase rate of the stock price within the past 10 minutes and the current stock price data with the search ID of the brand as a key. In general, stream data flows in at high speed from moment to moment, so that it is required to speed up the search processing included in the combination processing.

FIG. 7 is a block diagram illustrating an example of the configuration of an information processing apparatus that performs stream data combining processing. Referring to FIG. 7, the information processing apparatus includes a linear search unit 110, a linear storage unit 120, and a storage unit 130.

Based on the search data 100, the linear search unit 110 reads data linearly, that is, sequentially from the beginning, from the stream data sequence received in the past held in the storage unit 130, and the search ID included in the search data 100 Search for data having the same search ID. The linear storage unit 120 linearly writes newly arrived input data 200 in a storage unit that holds a stream data string. The linear storage unit 120 notifies the linear search unit 110 of the position of the newly written input data 200 in the stream data string. The linear search unit 110 determines where the oldest data is based on the notification from the linear storage unit 120.

According to such an information processing apparatus, since data is searched linearly, there is an advantage that it is easy to realize the requirement of maintaining the output order of the stream data string received in the past among the requirements of the above-described combination processing.

As an example, Non-Patent Document 1 describes a data stream processing device based on the same method as the information processing device shown in FIG.

Further, Patent Document 1 describes a system including a plurality of processing nodes and control nodes that process stream data in parallel.

JP 2007-251769 A

The entire disclosure of the above-mentioned patent documents and non-patent documents is incorporated by reference in this document. The following analysis was made by the present inventors.

The information processing apparatus shown in FIG. 7 searches for data linearly. Therefore, according to the information processing apparatus shown in FIG. 7, when the number of stream data sequences to be referred to increases among the stream data sequences received in the past, the search time increases, and there is a possibility that the stream data cannot be processed in real time. is there.

Further, in the system described in Patent Document 1, the control node stores the order in which the stream data is transferred and the transfer destination processing node as a pair, and processes the processing nodes in the order in which the stream data is transferred. Arrange the results. Thereby, the order of the stream data is maintained. However, according to such a configuration, it is necessary to provide a storage area in the control node for storing information composed of a pair of the transfer order of the stream data and the transfer destination processing node. In addition, the process in which the control node refers to the information and rearranges the processing results of the processing nodes may cause a bottleneck.

Therefore, it is desired to speed up the stream data combining process while maintaining the data order before and after the combining process. The objective of this invention is providing the information processing apparatus, the information processing method, and program which contribute to this request.

An information processing apparatus according to the first aspect of the present invention provides:
A write control unit, a plurality of data search units, and an output order control unit,
The write control unit periodically assigns input data included in stream data to the plurality of data search units in order,
The plurality of data search units store the assigned input data, search the input data stored first, including the same identifier as the identifier included in the search data, from the stored input data. Output,
The output order control unit sorts the input data output from the plurality of data search units in the order of the input data in the stream data according to the allocation status by the write control unit, and the plurality of data search units Are combined with the search data in the order after sorting and output.

An information processing method according to the second aspect of the present invention includes:
A computer periodically assigning input data included in stream data to a plurality of processing nodes in order;
The plurality of processing nodes that store the assigned input data, include the same identifier as the identifier included in the search data from the stored input data, and search for and output the first stored input data Sorting the input data output from the input data in the stream data according to the allocation status;
Combining the input data output from the plurality of processing nodes and the search data in the sorted order and outputting the combined data.

The program according to the third aspect of the present invention is:
A process of periodically assigning input data included in stream data to a plurality of processing nodes in order;
The plurality of processing nodes that store the assigned input data, include the same identifier as the identifier included in the search data from the stored input data, and search for and output the first stored input data Sorting the input data output from the input data in the stream data according to the allocation status;
A process of combining the input data output from the plurality of processing nodes and the search data in the sorted order and outputting the data is executed by a computer.
The program can be provided as a program product recorded on a non-transitory computer-readable storage medium.

According to the information processing apparatus, the information processing method, and the program according to the present invention, it is possible to speed up the stream data combining process while maintaining the data order before and after the combining process.

It is a block diagram which shows the structure of the information processing apparatus which concerns on one Embodiment as an example. It is a block diagram which shows the structure at the time of implement | achieving the information processing apparatus which concerns on one Embodiment as a computer and a some processing node as an example. It is a block diagram which shows the structure of the information processing apparatus which concerns on 1st Embodiment as an example. It is a figure which shows the data format in 1st Embodiment as an example. It is a block diagram which shows the structure of the data search part of the information processing apparatus which concerns on 1st Embodiment as an example. It is a block diagram which shows the structure of the output order control part of the information processing apparatus which concerns on 1st Embodiment as an example. It is a block diagram which shows the structure of the information processing apparatus which concerns on related technology as an example.

First, an outline of one embodiment will be described. Note that the reference numerals of the drawings attached to this summary are merely examples for facilitating understanding, and are not intended to limit the present invention to the illustrated embodiment.

FIG. 1 is a block diagram illustrating an example of a configuration of an information processing apparatus according to an embodiment. Referring to FIG. 1, the information processing apparatus includes a write control unit 30, a plurality of data search units 10-0, 10-1, ..., 10- (K-1), and an output order control unit 20. ing.

The write control unit 30 periodically inputs the input data included in the stream data to the plurality of data search units 10-0, 10-1,..., 10- (K−1) (that is, cyclic To). The plurality of data search units 10-0, 10-1,..., 10- (K-1) store the assigned input data and have the same identifier as the search data included in the stored input data. The input data including the identifier and stored first is searched for and output. The output order control unit 20 converts the input data output from the plurality of data search units 10-0, 10-1,..., 10- (K-1) into stream data according to the allocation status by the write control unit 30. Are sorted in the order of input data, and the input data and search data output from the plurality of data search units 10-0, 10-1,..., 10- (K-1) are combined and output in the order after sorting. To do.

The write control unit 30 may notify the output order control unit 20 of the data search unit that has finally assigned the input data included in the stream data. At this time, the output order control unit 20 inputs the input data output from the plurality of data search units 10-0, 10-1,..., 10- (K-1) in the stream data based on the notification. You may sort in the order of the data.

The write control unit 30 updates the count value in response to the arrival of the input data included in the stream data, and the data search units 10-0, 10-1,. There may be provided a counter that initializes the count value when the number reaches (ie, K). At this time, the write control unit 30 may allocate the input data included in the stream data to the plurality of data search units 10-0, 10-1, ..., 10- (K-1) based on the count value. .

Further, the write control unit 30 may notify the output order control unit 20 of the count value. At this time, the output order control unit 20 inputs the input data output from the plurality of data search units 10-0, 10-1,..., 10- (K-1) in the stream data based on the count value. You may sort in the order of the data.

Furthermore, the information processing apparatus (FIG. 1) according to an embodiment may be realized as a computer and a plurality of processing nodes. FIG. 2 is a block diagram showing an example of such a configuration.

Referring to FIG. 2, in the information processing method, the computer 60 cycles the input data included in the stream data in order with respect to a plurality of processing nodes 50-0, 50-1,..., 50- (K-1). And assigning the assigned input data, and searching for and outputting the first stored input data that includes the same identifier as the identifier contained in the search data from the stored input data Sorting the input data output from the plurality of processing nodes 50-0, 50-1,..., 50- (K-1) in the order of the input data in the stream data according to the allocation status, , 50- (K-1) and the input data output from the processing nodes 50-0, 50-1,..., 50- (K-1) are combined and output in the order after sorting.

According to the information processing apparatus and the information processing method, it is possible to speed up the stream data combining process while maintaining the data order before and after the combining process. This is because the write control unit 30 cyclically assigns the input data included in the stream data to the data search units 10-0, 10-1,..., 10- (K-1). This is because the input data output from the data search units 10-0, 10-1,..., 10- (K-1) can be easily sorted in the order of the input data in the stream data.

<Embodiment 1>
An information processing apparatus according to a first embodiment will be described with reference to the drawings. FIG. 3 is a block diagram illustrating an example of the configuration of the information processing apparatus according to the present embodiment. Referring to FIG. 3, the information processing apparatus includes K data search units 10-0, 10-1,..., 10- (K-1), an output order control unit 20, a write control unit 30, And a demultiplexer 40.

The data search unit 10-M (M = 0, 1,..., K-1) receives the input data 200 and the search data 100 as input, and stores the past stream data strings stored in the data search unit 10-M. The data having the same search ID as the search ID included in the search data 100 and its storage position are output. The input data 200 is stored in the data search unit 10-M as the latest data.

The output order control unit 20 receives K pieces of output data output from the data search units 10-0, 10-1,..., 10- (K-1), and outputs the output data according to the stored order. Output from oldest to newest. The output order control unit 20 refers to information notified from the write control unit 30 to determine whether the stored order is the oldest.

The write control unit 30 determines to which data search unit the input data is to be delivered based on the arrival notification of the input data 200 and passes the output to the demultiplexer 40 and the output order control unit 20. The write control unit 30 periodically issues a delivery instruction in the order of the data search units 10-0, 10-1, 10- (K-1). Since the delivery instruction is performed sequentially and periodically, according to this delivery instruction, it is possible to uniquely determine which data search unit holds the latest data. That is, it is possible to uniquely determine that the K data search units preceding the data search unit that holds the latest data hold the oldest data. As a specific implementation method, a counter with a period K that uses an arrival notification as a trigger for counting up can be used. However, it is sufficient if an integer string from 1 to K can be efficiently generated, and the present invention is not limited to a counter having a period K.

FIG. 4 is a diagram illustrating a data structure in the present embodiment as an example. Referring to FIG. 4, the search data 100 includes a search ID 400 and data. The input data 200 includes a search ID 400 and data. On the other hand, the output data 300 includes a search ID 400, search data 100 data, and input data 200 data. The search ID 400 included in the search data 100 and the search ID 400 included in the input data 200 have the same format (the same interpretation). On the other hand, the data stored in the search data 100 and the input data 200 may be in any format.

FIG. 5 is a block diagram illustrating an example of the configuration of the M-th (M = 0, 1,..., K−1) data search unit 10-M in the present embodiment. Referring to FIG. 5, data search unit 10 -M includes a data reading / comparing unit 11, a data writing unit 12, and a storage unit 13. For example, an element having a storage function such as a memory or a register can be used as the storage unit 13.

The data reading / comparing unit 11 refers to the search ID 400 included in the search data 100, and selects from the past stream data strings stored in the storage unit 13 (a pointer indicating the oldest position held internally) In this case, data having the same search ID 400 is searched in the oldest order, and if they match, the data is output. The data search may be any linear search as long as the order can be maintained.

The data writing unit 12 receives the input data 200 and writes the data at the oldest data position in the storage unit 13. When data is written, the data writing unit 12 notifies the data reading / comparing unit 11 of the writing. When the data reading / comparing unit 11 receives the writing notification from the data writing unit 12, the data reading / comparing unit 11 moves the pointer from the oldest position currently held to the next oldest position.

The storage unit 13 stores past stream data strings. Due to the periodicity of the delivery instruction by the write control unit 30 (that is, the cycle K), the data sequence included in the M-th data search unit 10-M stores a data sequence in which the remainder of K with respect to the arrival order is M. The At this time, the K data search units 10-0, 10-1,..., 10- (K-1) can be used to search K data strings in parallel from the oldest.

FIG. 6 is a block diagram illustrating an example of the configuration of the output order control unit 20 and the write control unit 30 in the present embodiment. Referring to FIG. 6, the output order control unit 20 includes a sort number adjustment unit 21, a sort unit 22, and a data combination unit 23. On the other hand, the write control unit 30 includes a counter 31.

The sort number adjustment unit 21 receives data having the same search ID as the search ID 400 included in the search data 100 from the K data search units 10-0, 10-1, ..., 10- (K-1). . Further, the sort number adjustment unit 21 specifies the data search unit that holds the oldest data based on the information of the data search unit that holds the latest data indicated by the write control unit 30. As a result, the sort number adjustment unit 21 labels (numbers) the output results of the K data search units 10-0, 10-1,..., 10- (K-1) in the oldest order. The received data is transferred to the sorting unit 22.

For example, when the write control unit 30 includes the counter 31 with the period K, when the data search unit 10-1 holding the latest data is instructed, the data search unit holding the oldest data Becomes the K-th previous data search unit, that is, the data search unit 10-2. As described above, since each data search unit 10-M searches the stream data sequence held by each data in the oldest order, the oldest data among the output data is output from the data search unit 10-2. It is understood that the old data is output in the order of data search units 10-3,..., 10- (K-1), 10-0, 10-1. Therefore, the sort number adjustment unit 21 assigns label 1 to the data output from the data search unit 10-2, assigns label 2 to the data output from the data search unit 10-3, and so on. The label K is assigned to the data output from the unit 10-1. Here, it is assumed that the output data arrives from all the data search units 10-M. However, if there is no output data, the label may be skipped or indicated to be invalid.

The sorting unit 22 receives the numbered data, sorts the data in the oldest order according to the number, and passes the sorted data to the data combining unit 23 in the oldest order. As an example, when the numbering as described above is performed, the sorting unit 22 may sort the data in ascending order of the numbers. Here, the sort number adjustment unit 21 and the sort unit 22 are separated from each other, but both may be realized as a single unit.

The data combining unit 23 combines the search ID and data included in the search data 100 and the data output from the sorting unit 22, that is, performs a combining process, and sequentially outputs the output data 300.

According to the information processing apparatus according to the present embodiment, the data having the same search ID, searched in parallel by the K data search units 10-0, 10-1,..., 10- (K-1). It becomes possible to perform the joining process from the oldest to the oldest.

In the present invention, the following modes are possible.
[Form 1]
The information processing apparatus according to the first aspect is as described above.
[Form 2]
The write control unit notifies the output order control unit of the data search unit that has finally assigned the input data included in the stream data;
The output order control unit may sort the input data output from the plurality of data search units in the order of the input data in the stream data based on the notification.
[Form 3]
The write control unit has a counter that updates a count value in response to arrival of input data included in the stream data, and initializes the count value when the number of updates reaches the number of the plurality of data search units. The input data included in the stream data may be assigned to the plurality of data search units based on the count value.
[Form 4]
The write control unit notifies the count value to the output order control unit,
The output order control unit may sort the input data output from the plurality of data search units in the order of the input data in the stream data based on the count value.
[Form 5]
The information processing method according to the second viewpoint is as described above.
[Form 6]
In the information processing method, the computer inputs input data output from the plurality of processing nodes based on information indicating a processing node to which input data included in the stream data is last assigned, in the stream data. You may sort in the order of the data.
[Form 7]
In the information processing method, the computer updates a count value in response to arrival of input data included in the stream data, and initializes the count value when the number of updates reaches the number of the plurality of data search units. Further comprising a step,
The computer may assign input data included in the stream data to the plurality of processing nodes based on the count value.
[Form 8]
In the information processing method, the computer may sort the input data output from the plurality of processing nodes in the order of the input data in the stream data based on the count value.
[Form 9]
The program is related to the third viewpoint.
[Mode 10]
The program sorts the input data output from the plurality of processing nodes in the order of the input data in the stream data based on the information indicating the processing node to which the input data included in the stream data is assigned last. Processing may be executed by the computer.
[Form 11]
The program updates the count value in response to arrival of input data included in the stream data, and initializes the count value when the number of updates reaches the number of the plurality of data search units;
Processing for assigning input data included in the stream data to the plurality of processing nodes based on the count value may be executed by the computer.
[Form 12]
The program may cause the computer to execute a process of sorting the input data output from the plurality of processing nodes in the order of the input data in the stream data based on the count value.

It should be noted that the disclosures of the above patent documents and non-patent documents are incorporated herein by reference. Within the scope of the entire disclosure (including claims) of the present invention, the embodiment can be changed and adjusted based on the basic technical concept. Further, various combinations or selections of various disclosed elements (including each element of each claim, each element of each embodiment, each element of each drawing, etc.) are possible within the framework of the entire disclosure of the present invention. is there. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, with respect to the numerical ranges described in this document, any numerical value or small range included in the range should be construed as being specifically described even if there is no specific description.

10-0, 10-1,..., 10- (K-1) Data search unit 11 Data read / comparison unit 12 Data write unit 13 Storage unit 20 Output order control unit 21 Sort number adjustment unit 22 Sort unit 23 Data combination Unit 30 write control unit 31 counter 40 demultiplexer 50-0, 50-1,..., 50- (K-1) processing node 60 computer 100 search data 110 linear search unit 120 linear storage unit 130 storage unit 200 input data 300 Output data 400 Search ID

Claims

A write control unit, a plurality of data search units, and an output order control unit,
The write control unit periodically assigns input data included in stream data to the plurality of data search units in order,
The plurality of data search units store the assigned input data, search the input data stored first, including the same identifier as the identifier included in the search data, from the stored input data. Output,
The output order control unit sorts the input data output from the plurality of data search units in the order of the input data in the stream data according to the allocation status by the write control unit, and the plurality of data search units An information processing apparatus that combines and outputs the input data output from and the search data in the sorted order.
The write control unit notifies the output order control unit of the data search unit that has finally assigned the input data included in the stream data;
The information processing apparatus according to claim 1, wherein the output order control unit sorts input data output from the plurality of data search units in an order of input data in the stream data based on the notification.
The write control unit has a counter that updates a count value in response to arrival of input data included in the stream data, and initializes the count value when the number of updates reaches the number of the plurality of data search units. The information processing apparatus according to claim 1, wherein input data included in the stream data is allocated to the plurality of data search units based on the count value.
The write control unit notifies the count value to the output order control unit,
The information processing apparatus according to claim 3, wherein the output order control unit sorts input data output from the plurality of data search units in an order of input data in the stream data based on the count value.
A computer periodically assigning input data included in stream data to a plurality of processing nodes in order;
The plurality of processing nodes that store the assigned input data, include the same identifier as the identifier included in the search data from the stored input data, and search for and output the first stored input data Sorting the input data output from the input data in the stream data according to the allocation status;
Combining the input data output from the plurality of processing nodes and the search data in the sorted order and outputting the combined data.
The computer sorts the input data output from the plurality of processing nodes in the order of the input data in the stream data based on information indicating a processing node that has finally assigned the input data included in the stream data. The information processing method according to claim 5.
The computer further includes a step of updating the count value in response to arrival of input data included in the stream data, and initializing the count value when the number of updates reaches the number of the plurality of data search units,
The information processing method according to claim 5, wherein the computer assigns input data included in the stream data to the plurality of processing nodes based on the count value.
The information processing method according to claim 7, wherein the computer sorts input data output from the plurality of processing nodes in an order of input data in the stream data based on the count value.
A process of periodically assigning input data included in stream data to a plurality of processing nodes in order;
The plurality of processing nodes that store the assigned input data, include the same identifier as the identifier included in the search data from the stored input data, and search for and output the first stored input data Sorting the input data output from the input data in the stream data according to the allocation status;
A program that causes a computer to execute a process of combining input data output from the plurality of processing nodes and the search data in the sorted order and outputting the combined data.
The process of sorting the input data output from the plurality of processing nodes in the order of the input data in the stream data, based on the information indicating the processing node that lastly assigned the input data included in the stream data, The program according to claim 9, which is executed by a computer.
Updating the count value in response to arrival of input data included in the stream data, and initializing the count value when the number of updates reaches the number of the plurality of data search units;
The program according to claim 9 or 10, which causes the computer to execute a process of allocating input data included in the stream data to the plurality of processing nodes based on the count value.
12. The program according to claim 11, wherein the computer executes a process of sorting the input data output from the plurality of processing nodes in the order of the input data in the stream data based on the count value.