JP2011141587A - Distributed processing system, distributed processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a response time to data having a lot of single information amount uploaded to a network. <P>SOLUTION: When receiving data, the reception response device 10 of a distributed processing system 1 transmits the received data to a division integration device 20. A fragment data generation part 22 of the division integration device 20 performs cutout in a unit processable when a processor performs data processing by a prescribed data size and pattern retrieval, generates fragment data, and transmits them to a queue management device 30. A plurality of processors 40 each receive the fragment data from the queue management device 30, each execute the data processing to generate fragment result data, and each transmit them to the division integration device 20. The division integration device 20 having obtained the fragment result data from each processor 40 integrates all the fragment result data, generates result data, and transmits them to a user terminal 2 through the reception response device 10. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a distributed processing system, a distributed processing method, and a program for upload data.

  When providing a service on the Web, the amount of data for a single piece of data is enormous, and when some processing is performed, the processing result is provided to the user, but in some cases it cannot be processed in real time. Exists. For example, Japanese-English translation processing, Japanese dependency analysis processing, attention scene extraction from video, and the like. Specifically, in the case of Japanese dependency analysis processing, there is a case where the dependency analysis processing takes 100 seconds for a certain design document 16000 lines (700 KB). In this case, if this process is executed as a service via a Web browser, a browser timeout occurs before 100 seconds. Therefore, conventionally, the following measures have been taken in such a case.

  (1) Prohibit input of data with a large amount of information. There are a large number of service systems that provide such restrictions, but there is a problem that the service itself cannot be used in the first place if the restrictions required of the user are too strict.

  (2) An asynchronous result confirmation interface is provided. In preparation for cases where users cannot respond within the browser timeout period, new processing groups such as notifying the user by e-mail after processing is completed, for example, e-mail address registration, e-mail transmission, error resending, result progress status inquiry Prepare etc. In this case, there is a problem that the processing configuration becomes complicated in a non-essential part of the system.

  (3) The processing result is generated in a storage area or the like where user authentication is possible, and the user confirms that the result is generated at an appropriate time. In this case, in order to obtain a result that the user does not know when it is generated, it is necessary to confirm the result many times, and there is a problem that the burden on the user increases.

  On the other hand, there is a MapReduce model as a technique for speeding up processing by utilizing a large number of PC groups (see Non-Patent Document 1). The MapReduce model stores, for example, a detailed data group (Web page group) obtained by crawling the Web on a distributed file system (GFS), and performs morphological analysis processing and word processing on a machine that provides each distributed file system. The counting process is executed in parallel. Thereby, it is possible to create a transposed file in a short time for a huge file group having a total capacity of 10 TB.

  Also, a distributed configuration has been proposed as a technique for improving the system throughput (see Patent Document 1). In this distributed configuration, there are a plurality of reception queues and a plurality of processing units, a plurality of data are received by a plurality of reception queues, and a plurality of received data are distributed and processed by each processing unit. Thus, the throughput of the entire system can be improved. This system is called a write-once reference type DMS (Data Management System) or simply DMS.

IT Glossary, “MapReduce”, [online], [Search on December 20, 2009], Internet <URL: http://e-words.jp/w/MapReduce.html>

JP 2008-83808 A

  However, the MapReduce model, which is a technique described in Non-Patent Document 1, requires that it takes time to schedule which machine to execute processing before calculation, and on the distributed file system. There are strong restrictions such as the need to register data, and it is suitable for batch processing and not for real-time processing. Therefore, it cannot be applied to speeding up the targeted processing in response to data uploaded from the user. Basically, it corresponds to the case where there are many small files such as Web page data. For the input of a single information-rich file as the object of the present invention, data Even if the division is performed only with the size of the file, it is difficult to execute the distributed processing while maintaining the semantic matching in the vicinity of the division.

  In addition, the write-once reference type DMS, which is a technique described in Patent Document 1, aims to improve the throughput of the entire system when receiving a plurality of data. It was not possible to speed up the response to a large amount of data.

  As described above, there are technologies for reducing the processing time of batch processing and technologies for improving the processing throughput for a plurality of pieces of data that have already been divided. However, the upload data has a large amount of information from the user. In response, the distributed processing was executed while maintaining the semantic consistency of the data, and the response time of the target processing could not be shortened.

  The present invention has been made in view of such a background, and the present invention provides a distributed processing system, a distributed processing method, and a method capable of speeding up a response to a single data having a large amount of information. The purpose is to provide a program.

  In order to solve the above-described problem, the invention according to claim 1 is: (1) accepting upload data from a user terminal via a communication network, and synchronously or asynchronously processing result data of the accepted upload data. An acceptance response device that replies to the user terminal; and (2) a plurality of pieces of fragment data are generated by dividing the upload data acquired from the acceptance response device, and the generated pieces of fragment data are subjected to data processing. (3) a fragmentation integration device that integrates a plurality of fragment result data to generate the result data; and (3) obtains the fragment data from the fragmentation integration device and stores it as a queue, from one of a plurality of processing devices One or more queue management devices that transmit the fragment data to the processing device in response to the request, and (4) the fragment data A plurality of processing devices that are acquired from the queue management device, process the acquired fragment data, and transmit the fragment data as the fragment result data to the fragmentation integration device. A device receives the upload data from the user terminal via the communication network, and obtains the upload data from the data reception unit as a file, and stores the filed data in the data storage unit After that, a file processing unit that generates a startup instruction message for starting up the division integration device, a startup instruction message generated by the file processing unit and the filed data are transmitted to the division integration device, and the division integration device The result data acquired from the network via the communication network A transmission control unit that transmits the data to the user terminal, and the division integration device reads the data transmitted from the acceptance response device in order from the beginning of the data, and the data reading unit reads the data. Fragment data generation that determines the type of data and generates the plurality of fragment data by cutting out in units that can be processed when the processing device processes the data according to the determined data type Each of the plurality of fragment data is assigned a sequential ID and transmitted to the one or more queue management devices in the order in which one of the plurality of fragment data is generated by the fragment data generation unit and the fragment data generation unit. A data management unit that receives the fragment result data transmitted from the plurality of processing devices, and associates the fragment result data with the ID attached to the fragment data A fragment result data acquisition unit that determines whether or not all of the fragment result data has been acquired based on a sequential ID attached to the fragment result data, and the fragment result data acquisition unit includes all of the fragment result data The fragment result data integration unit that integrates the fragment result data in the order of the IDs and generates the result data using the sequential ID attached to the fragment result data, and the fragment A data storage unit that stores the result data generated by the result data integration unit, and a result data output unit that transmits the result data stored in the data storage unit to the reception response device, A queue management device that receives the fragment data transmitted from the dividing and integrating device and stores it in a queue storage unit as a queue. A fragment management unit that transmits the fragment data stored in the queue storage unit to the processing device based on the fragment data acquisition request from the processing device, and the fragment data received by the queue reception unit The queue storage unit to be stored, and the processing device transmits the fragment data acquisition request to the queue management device and acquires the fragment data from the queue management device, and the fragment A data processing unit that executes predetermined data processing on the fragment data acquired by the data acquisition unit, and the fragment result data that is a result of data processing performed by the data processing unit, is attached to the fragment data. A fragment result data output unit that adds a sequential ID associated with the ID and transmits the fragment ID to the fragmentation integration device. It was distributed processing system.

  The invention according to claim 6 is: (1) accepting upload data from a user terminal via a communication network, and returning the data processing result data for the accepted upload data to the user terminal synchronously or asynchronously. An acceptance response device; and (2) a plurality of fragment result data obtained by dividing the upload data acquired from the acceptance response device to generate a plurality of fragment data, and the plurality of the generated fragment data are subjected to data processing. (3) acquiring the fragment data from the fragmentation integration device and storing it as a queue, and requesting the fragment data in response to a request from one of a plurality of processing devices. One or more queue management devices that transmit to the processing device, and (4) whether the fragment data is the queue management device A distributed processing method for use in a distributed processing system comprising: the plurality of processing devices that acquire, process the acquired fragment data, and transmit the fragment data as the fragment result data to the fragmentation integration device, A responding device comprising a data storage unit in which the upload data is stored as a file, the step of receiving the upload data from the user terminal via the communication network, and acquiring the received upload data to obtain a file And generating the activation instruction message for starting the division integration device after storing the filed data in the data storage unit, and the division integration device that generates the activation instruction message and the filed data. And sending to the split A step in which the device reads the data transmitted from the acceptance response device in order from the beginning of the data, determines the type of the read data, and the processing device sets the data according to the determined type of data; Generating the plurality of fragment data by cutting out in units that can be processed when data is processed, and sequentially generating each of the plurality of fragment data in the order in which one of the plurality of fragment data is generated. And a step of transmitting to the one or more queue management devices with an ID, wherein the queue management device stores the fragment data transmitted from the division integration device as a queue. Including the step of accepting the fragment data transmitted from the fragmentation integration device and storing it as a queue in a queue storage unit; Transmitting the fragment data stored in the queue storage unit to the processing device based on the fragment data acquisition request from the processing device, and the processing device sends the queue management device to the queue management device. Transmitting a fragment data acquisition request and acquiring the fragment data from the queue management device; executing predetermined data processing on the acquired fragment data; and the data processed result being A step of attaching a sequential ID associated with the ID attached to the fragment data to the fragment result data and transmitting the fragment result data to the fragmentation integration device, wherein the fragmentation integration device performs the fragment result data A data storage unit for storing the result data integrated, and receiving the fragment result data transmitted from the plurality of processing devices, Determining whether or not all of the fragment result data has been acquired based on the sequential ID attached to the fragment result data in association with the ID attached to the fragment data; If it is determined that fragment result data has been acquired, using the sequential ID attached to the fragment result data, integrating the fragment result data in the order of the IDs to generate the result data; and the fragment result Storing the result data generated by the data integration unit in the data storage unit, and transmitting the result data stored in the data storage unit to the reception response device; The response device sends the result data acquired from the division integration device to the user terminal via the communication network. It was distributed processing method comprising the steps performed.

  Thus, according to the present invention, even if there is a single large amount of information, if a plurality of processing devices are prepared, the response to processing requests from the user terminal can be speeded up. Can do. Therefore, the result can be returned to the user terminal within the timeout time of the browser. In addition, it is possible to omit the construction of an accompanying system (system for e-mail address registration, e-mail transmission, etc.) such as notification of the processing result after the fact.

  According to a second aspect of the present invention, the acceptance response device generates the activation instruction message when the data acceptance unit starts accepting the upload data, and sends the activation instruction message to the division integration device via the transmission control unit. Each time a predetermined amount of the upload data received by the data receiving unit is acquired and divided into a file, the filed data is sequentially stored in the data storage unit, and the data is stored in the data storage unit. Each time the data is stored in the storage unit, the data storage unit further includes a sequential file processing unit that transmits the data to the dividing and integrating device via the transmission control unit. The dividing and integrating device, the data reading unit, and the reception response device. Each time the data is read, the data is transferred to the fragment data generation unit, and the fragment data generation unit generates the fragment data. It was distributed processing system according to claim 1 that.

  In this way, according to the present invention, the data upload processing from the user terminal to the reception response device, the fragment data generation processing by the fragmentation integration device, and the fragment data transmission processing to the queue management device are performed in parallel. Therefore, the response (response time) can be further shortened.

  According to a third aspect of the present invention, the fragment data generation unit of the fragmentation integration device reads the data up to a predetermined amount when the type of the data read by the data reading unit is Japanese text data, Fragment as a portion that can be processed when the processing device processes the data from the end of the read data to the front or the back, detecting a punctuation character and the following line feed code and up to the detected point The distributed processing system according to claim 1 or 2, further comprising a Japanese text fragment data generation unit cut out as data.

  In this way, when Japanese text data is uploaded to the acceptance response device, the Japanese text fragment data generation unit of the fragmentation integration device searches for the data size, punctuation characters, and the following line feed code. Fragment data can be generated by cutting out in units that can be processed when the processing device processes data.

  According to a fourth aspect of the present invention, the fragment data generation unit of the fragmentation integration apparatus reads the data up to a predetermined amount when the type of the data read by the data reading unit is English text data, and reads the data From the end of the data, a period and subsequent line feed code are detected forward or backward from the end of the data, and up to the detected point as fragment data as a portion that can be processed when the processing unit processes the data. The distributed processing system according to any one of claims 1 to 3, further comprising an English text fragment data generation unit to be cut out.

  In this way, when English text data is uploaded to the reception response device, the English text fragment data generation unit of the fragmentation integration device searches for the data size, the period, and the line feed code that follows it, and the processing device Can generate fragment data by cutting out in units that can be processed.

  The invention according to claim 5, in the case where the fragment data generation unit of the fragmentation integration device is video data in which the type of the data read by the data reading unit is configured in GOP (Group Of Pictures) units, 5. The distributed processing system according to claim 1, further comprising a video fragment data generation unit that generates the fragment data by cutting out the video data for each predetermined number of GOPs. did.

  In this way, when a single piece of video data with a large amount of information is input, by preparing a plurality of processing devices, response (response time) can be shortened in response to a processing request from the user. It becomes possible to plan.

  The invention according to claim 7 is a program for causing a computer to execute the distributed processing method according to claim 6.

  According to such a program, the distributed processing method according to the sixth aspect can be realized by a general computer.

  According to the present invention, it is possible to provide a distributed processing system, a distributed processing method, and a program capable of speeding up the response to data having a large amount of information.

It is a functional block diagram which shows the structural example of the distributed processing system which concerns on the 1st Embodiment of this invention. It is a sequence diagram which shows the flow of the distributed processing by the distributed processing system which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the flow of the fragment | piece data generation process which the text fragment | piece data generation part of the division | segmentation integration apparatus which concerns on the 1st Embodiment of this invention performs. It is a figure which shows an example of the Japanese text data input into the distributed processing system which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the result data which the distributed processing system which concerns on the 1st Embodiment of this invention produces | generates. It is a figure which shows an example of the fragment data which the text fragment data generation part of the division | segmentation integration apparatus which concerns on the 1st Embodiment of this invention produces | generates. It is a sequence diagram which shows the flow of the distributed processing of the distributed processing system which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the flow of the fragment | piece data generation process which the video fragment data generation part of the division | segmentation integration apparatus which concerns on the 3rd Embodiment of this invention performs. It is a block diagram for demonstrating the modification of the distributed processing system which concerns on embodiment of this invention.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described in detail with reference to the drawings as appropriate.

<< First Embodiment >>
FIG. 1 is a functional block diagram showing a configuration example of a distributed processing system 1 according to the first embodiment of the present invention.

  First, an overview of distributed processing in the distributed processing system 1 according to the first embodiment of the present invention will be described with reference to FIG. First, the acceptance response device 10 accepts data upload from the user terminal 2 via the network 3. The data uploaded at this time is, for example, text data having a large amount of information as shown in FIG. The reception response device 10 transmits the received single data to the dividing and integrating device 20, and the dividing and integrating device 20 cuts out in units that can be processed when the processing device performs data processing by searching for a predetermined data size and pattern. Then, fragment data (see FIG. 6 described later) is generated and transmitted to one or more queue management devices 30. Each of the plurality of processing devices 40 receives the fragment data from the queue management device 30 and executes data processing (for example, Japanese-English translation processing) to generate fragment result data. The fragmentation and integration device 20 that has acquired the fragment result data from each processing device 40 integrates all the fragment result data, generates result data (see FIG. 5 to be described later), and receives the user terminal via the reception response device 10. 2 to send.

  As described above, the dividing and integrating device 20 divides the uploaded single data, distributes the data by the plurality of processing devices 40, performs the data processing, and then integrates the data, whereby the first of the present invention. According to the distributed processing system 1 according to the embodiment, the response (response time) to uploaded data can be shortened.

Next, the configuration of the distributed processing system 1 according to the first embodiment of the present invention will be specifically described.
The distributed processing system 1 acquires data (upload data) received from the user terminal 2 via the network 3, and acquires the data received by the reception response device 10, and divides the data to obtain fragment data. , And further integrate the processed fragment result data, a plurality of processing devices 40 (40a, 40b, 40c,...) That acquire fragment data and perform data processing, and the fragmentation integration device 20 One or more queue management devices 30 (30a, 30b,...) That acquire fragment data output from the storage device and store it as a queue and transmit the fragment data to the processing device 40 in response to a request from the processing device 40. Consists of.

  In addition, (1) between the reception response device 10 and the division integration device 20, (2) the division integration device 20, the queue management device 30 (30a, 30b,...) And the processing device 40 (40a, 40b, 40c,. ..) And (3) the queue management device 30 (30a, 30b,...) And the processing device 40 (40a, 40b, 40c) are connected by a network or a dedicated line (not shown). The number of queue management devices 30 is one or more queue management devices 30 (30a, 30b,...), And the number of processing devices 40 is two or more processing devices 40 (40a, 40b, 40c,...). ).

<Reception response device>
The reception response device 10 receives single data (upload data) transmitted from the user terminal 2 via the network 3. The reception response device 10 includes a data reception unit 11, a file processing unit 12, a transmission control unit 13, and a data storage unit 14. Although not shown here, the reception response device 10 is a computer including an input unit that controls input of various data, an output unit that controls output, a CPU (Central Processing Unit), and a communication interface. It is realized by.

  The data reception unit 11 receives single data (upload data) from the user terminal 2 via the network 3 or an input unit (not shown). Then, the received data is delivered to the file processing unit 12.

  The file processing unit 12 converts the data acquired from the data receiving unit 11 into a file and stores it in the data storage unit 14. Further, the file processing unit 12 transmits an activation instruction message to the dividing and integrating device 20 via the transmission control unit 13 when the data upload is completed. Then, the file processing unit 12 acquires result data that has been subjected to data processing and integration from the dividing and integrating device 20, and transmits the result data to the user terminal 2 via the transmission control unit 13. The sequential file processing unit 121 included in the file processing unit 12 illustrated in FIG. 1 is included in the distributed processing system 1a according to the second embodiment to be described later, and thus description thereof is omitted here.

  The transmission control unit 13 performs control to transmit the data stored in the data storage unit 14 to the division integration device 20 according to an instruction from the file processing unit 12. The transmission control unit 13 performs control to transmit the result data acquired by the file processing unit 12 to the user terminal 2.

  The data storage unit 14 is a storage unit that stores data received by the data receiving unit 11 and is realized by a RAM (Random Access Memory), a HDD (Hard Disk Drive), or the like.

  The data receiving unit 11, the file processing unit 12, and the transmission control unit 13 are realized by a program execution process by the CPU or dedicated hardware. When the program is executed by the CPU, the data storage unit 14 stores these programs.

<Partition integration device>
The division integration device 20 cuts out the data received from the reception response device 10 in units that can be processed when the processing device performs data processing, and generates fragment data. And the division | segmentation integration apparatus 20 integrates | segments the fragment result data after processing by each processing apparatus 40 (40a, 40b, 40c, ...), and produces | generates one processed result data. The fragmentation integration device 20 includes a data reading unit 21, a fragment data generation unit 22, a data management unit 23, a fragment result data acquisition unit 24, a fragment result data integration unit 25, a result data output unit 26, a data And a storage unit 27. Although illustration is omitted here, the division integration device 20 is realized by a computer including an input unit that controls input of various data, an output unit that controls output, and a CPU.

  The data reading unit 21 reads the data transmitted by the acceptance response device 10. Then, the read data is delivered to the fragment data generation unit 22.

  The fragment data generation unit 22 determines the type of the delivered data, and generates fragment data by cutting out in units that can be processed when the processing device processes the data. Here, the extraction in units that can be processed when data is processed by the processing device, for example, if the data is Japanese text data, it is delimited by punctuation marks and line feeds, and the meaning is understood as Japanese. This means that the sentence is cut out at a break between the sentences (see FIGS. 3 and 6). In addition, in the case of video data such as mpeg2 to be described later, it refers to segmentation of video in units of several minimum unit structures constituting a moving image called GOP (Group Of Pictures) (see FIG. 8).

  The fragment data generation unit 22 includes a text fragment data generation unit 221 and a video fragment data generation unit 222. Note that the video fragment data generation unit 222 is provided in the distributed processing system 1b according to the third embodiment to be described later, and a description thereof is omitted here.

  When the data type is text data, the text fragment data generation unit 221 first reads a predetermined number of bytes, and forward (or backward) a punctuation character from the last point of the read data. Then, it is detected whether or not there is a subsequent line feed code, and processing up to the detected point is cut out as fragment data as a portion that can be processed when the processing device performs data processing. In the present embodiment, the forward means processing toward the data side to be read in the future after viewing from a certain data, and the backward means reading before the data from a certain data. Processing toward the data side. The fragment data generation processing by the text fragment data generation unit 221 will be described in detail with reference to FIG.

  The data management unit 23 assigns a sequential ID to each piece of fragment data in the order in which the piece data is generated by the piece data generation unit 22, and the last piece of data cut out by the piece data generation unit 22 Is attached (hereinafter referred to as “last flag”). Then, the data management unit 23 transmits the fragment data to which the ID is attached to the queue management device 30. At this time, if there are a plurality of queue management devices 30, considering the load distribution of the reception queues, etc., a distribution algorithm such as a round robin method is used for the plurality of queue management devices 30 (30 a, 30 b,...). It is also possible to transmit fragment data in a distributed manner. By doing so, it is possible to reduce the processing load of the queue management device 30 and prevent the process waiting queue from stagnation.

  The fragment result data acquisition unit 24 acquires the fragment result data from the processing device 40 (40a, 40b, 40c,...), And the data management unit 23 attaches the fragment data to the ID and the ID carried over to the fragment result data as a file. As a name, the fragment result data is stored in the data storage unit 27. Further, the fragment result data acquisition unit 24 determines whether or not the fragment result data corresponding to all the fragment data has been acquired, using the ID attached to the fragment result data.

  The fragment result data integration unit 25 integrates each fragment result data by merging in the order of IDs attached to the fragment result data when the fragment result data acquisition unit 24 acquires all the fragment result data. , Generate result data. Further, the fragment result data integration unit 25 stores the generated result data in the data storage unit 27.

  The result data output unit 26 generates a data processing completion notification triggered by the generation of the result data by the fragment result data integration unit 25, and transmits the result data to the reception response device 10 together with the completion notification.

  The data storage unit 27 includes data acquired from the reception response device 10, fragment data generated by the fragment data generation unit 22, fragment result data acquired by the fragment result data acquisition unit 24, and results generated by the fragment result data integration unit 25. This is a storage means for storing data and the like, and is realized by a RAM, HDD or the like.

  The data reading unit 21, the fragment data generation unit 22, the data management unit 23, the fragment result data acquisition unit 24, the fragment result data integration unit 25, and the result data output unit 26 are a program execution process by the CPU or dedicated hardware. It is realized by. When the program is executed by the CPU, the data storage unit 27 stores these programs.

<Queue management device>
The queue management device 30 (30a, 30b,...) Manages and stores the fragment data transmitted from the dividing and integrating device 20 as a queue, and in response to a request from the processing device 40 (40a, 40b, 40c,...) The fragment data is transmitted to the requested processing apparatus 40. The queue management device 30 includes a queue reception unit 31, a queue management unit 32, and a queue storage unit 33. Although not shown here, the queue management device 30 is realized by a computer including an input unit that controls input of various data, an output unit that controls output, and a CPU.

  The queue reception unit 31 acquires the fragment data with the ID transmitted from the dividing and integrating device 20 and stores it in the queue storage unit 33 as a queue. Further, when there is a request for obtaining fragment data from the processing device 40 (40a, 40b, 40c,...), The queue management unit 32 transmits the fragment data stored in the queue storage unit 33 to the processing device 40. .

  The queue storage unit 33 is a storage unit that stores the fragment data acquired by the queue reception unit 31, and is realized by a RAM, an HDD, or the like.

<Processing device>
The processing device 40 acquires fragment data from the queue management device 30 and executes predetermined data processing on the acquired fragment data. The processing device 40 includes a queue selection unit 41, a fragment data acquisition unit 42, a data processing unit 43, and a fragment result data output unit 44. Although not shown here, the processing device 40 is a computer that includes an input unit that controls input of various data, an output unit that controls output, a storage unit that stores various data, and a CPU. Realized.

  The queue selection unit 41 selects the queue management device 30 from which fragment data is acquired. The selection of the queue management device 30 at this time is performed using a distributed algorithm such as a round robin method.

  The fragment data acquisition unit 42 transmits a fragment data acquisition request to the queue management device 30 selected by the queue selection unit 41 and acquires fragment data from the queue management device 30.

  The data processing unit 43 performs predetermined data processing on the fragment data acquired by the fragment data acquisition unit 42. For example, Japanese-English translation processing, Japanese dependency analysis processing, etc. are executed. For video data described later, cut point detection processing, attention scene detection processing, and the like are executed.

  The fragment result data output unit 44 outputs fragment result data, which is a result of data processing performed by the data processing unit 43, to the dividing and integrating device 20. At this time, the sequential ID assigned to the fragment data by the data management unit 23 of the dividing and integrating device 20 is directly inherited to the fragment result data.

  As described above, the reception response device 10, the division integration device 20, the queue management device 30 (30a, 30b,...), And the processing device 40 (40a, 40b, 40c,. These may be realized by separate computers (devices), or may be realized as a single computer.

<Operation procedure>
Next, the operation procedure of the distributed processing system 1 according to the first embodiment of the present invention will be described in detail with reference to FIG. 1 and FIG. FIG. 2 is a sequence diagram showing the flow of distributed processing by the distributed processing system 1 according to the first embodiment of the present invention.

  First, when data is transmitted from the user terminal 2 via the network 3 (step S101), the data reception unit 11 of the reception response device 10 receives the data (step S102). Then, the file processing unit 12 of the reception response device 10 determines whether or not the data reception by the data reception unit 11 is completed (Step S103). If data reception has not been completed (that is, if data is being received) (step S103 → No), the process returns to step S102, and the data receiving unit 11 continues to receive data. On the other hand, when the data reception is completed (step S103 → Yes), the file processing unit 12 stores the data received by the data receiving unit 11 in the data storage unit 14 as one file.

  Further, the file processing unit 12 transmits an activation instruction message to the dividing and integrating device 20 via the transmission control unit 13 when the data reception (upload) is completed. And the transmission control part 13 transmits the data memorize | stored in the data storage part 14 to the division | segmentation integration apparatus 20 (step S104).

  Next, the division integration device 20 activated upon reception of the activation instruction message from the reception response device 10 starts data reading by the data reading unit 21 (step S105). Then, the data reading unit 21 delivers the data to the fragment data generation unit 22 while reading the data.

  Subsequently, the fragment data generation unit 22 determines the type of the delivered data, and performs fragment data generation processing by cutting out in units that can be processed when the processing apparatus processes the data (step S106). Details of the fragment data generation processing by the fragment data generation unit 22 will be described later with reference to FIG.

  The data management unit 23 receives fragment data every time fragment data is generated by the fragment data generation unit 22, assigns sequential IDs in the order of generation of each fragment data, and the fragment data generation unit 22 A flag (last flag) that can be identified as the last is added to the last fragment data that has been cut out. Then, when there are a plurality of queue management devices 30, the data management unit 23 determines the queue management device 30 as the data output destination using the round robin method or the like, and sends the determined queue management device 30 to the determined queue management device 30. The fragment data is transmitted (step S107). Subsequently, the data management unit 23 cuts out all of the data read by the data reading unit 21 and determines whether all of the fragment data has been transmitted (step S108). If there is still data being read and all of the fragment data has not been transmitted (step S108 → No), the process returns to step S105, and the data reading unit 21 continues to read the data. On the other hand, if all of the fragment data has been transmitted (step S108 → Yes), the process is terminated.

  Next, the queue receiving unit 31 of the queue management device 30 receives the fragment data with the ID from the dividing and integrating device 20 (step S109) and stores it as a queue in the queue storage unit 33. In response to a fragment data acquisition request from the processing device 40 (40a, 40b, 40c,...), The queue management unit 32 converts the fragment data stored in the queue storage unit 33 into the processing device 40 (40a, 40b,. 40c,...) (Step S110).

  Subsequently, the queue selection unit 41 of the processing device 40 selects the queue management device 30 from which the fragment data is acquired using the round robin method or the like (step S111). Then, the fragment data acquisition unit 42 of the processing device 40 transmits a fragment data acquisition request to the queue management device 30 selected by the queue selection unit 41, and acquires fragment data from the queue management device 30 (step S112). ). Next, the data processing unit 43 performs predetermined data processing on the fragment data acquired by the fragment data acquisition unit 42 (step S113). Then, the fragment result data output unit 44 transmits the fragment result data, which is the processing result of the data processing unit 43, to the dividing and integrating device 20 (step S114).

  For example, the processing device 40 acquires fragment data of Japanese text from the queue management device 30 and performs a Japanese-English translation process in the data processing unit 43. Then, the fragment result data obtained as a result of the data processing is output to the dividing and integrating device 20 with the ID attached to the fragment data attached. Further, the processing device 40 different from the one processing device 40 performs the same processing, and each fragment result data subjected to data processing in each processing device 40 (40a, 40b, 40c,...) Is divided and integrated. 20 is transmitted.

  In this way, data processing such as Japanese-English translation processing, which requires processing time, is divided into a plurality of pieces of fragment data and then distributed and executed in a plurality of processing devices 40, thereby reducing the overall processing time. Is possible.

  Next, the fragment result data acquisition unit 24 of the division integration device 20 acquires fragment result data from each processing device 40 (40a, 40b, 40c,...) (Step S115). The fragment result data integration unit 25 converts the ID attached to the fragment result data acquired by the fragment result data acquisition unit 24 into a file name or the like and stores the file in the data storage unit 27. Further, the fragment result data integration unit 25 determines whether or not the last flag is given to the acquired fragment result data. Then, when the fragment result data integration unit 25 acquires the fragment data after acquiring the fragment result data to which the last flag is added, the fragment result data integration unit 25 converts the ID assigned to the fragment data into a file name or the like as a file. At the same time, it is determined whether or not all pieces of fragment data have been acquired (step S116).

  If there is fragment data that has not yet been acquired (step S116 → No), the process returns to step S115 to continue acquiring fragment data. On the other hand, when all pieces of fragment data have been acquired (step S116 → Yes), the acquisition process ends, and the fragment result data integration unit 25 merges the fragment result data by merging in the order of IDs attached to the files. Result data is generated (step S117), and the generated result data is stored in the data storage unit 27.

  Subsequently, the result data output unit 26 generates a processing completion notification and transmits the processing completion notification to the reception response device 10 (step S118). Next, the data reception unit 11 of the reception response device 10 that has received the processing completion notification transmits the processing completion notification to the user terminal 2 via the transmission control unit 13 (that is, the result data is asynchronously transmitted to the user terminal 2). Reply) (step S119). In step S118, when the result data output unit 26 of the division integration device 20 transmits the process completion notification to the reception response device 10, the result data is transmitted together with the processing completion notification to the reception response device 10. Then, it may be transmitted from the acceptance response device 10 to the user terminal 2 (that is, the result data is returned to the user terminal 2 synchronously). Then, the user terminal 2 receives the process completion notification (result data) and ends the process (step S120).

(Fragment data generation process)
Next, the fragment data generation process performed by the fragment data generation unit 22 of the division integration device 20 in step S106 of FIG. 2 will be described. FIG. 3 is a flowchart showing a flow of fragment data generation processing performed by the text fragment data generation unit 221 of the fragmentation integration apparatus 20 according to the first embodiment of the present invention. In the present embodiment, the Japanese text data shown in FIG. 4 is uploaded, and this data is subjected to a Japanese-English translation process by the processing device 40, thereby converting it into English text data shown in FIG. Will be described. In this case, the text fragment data generation unit 221 provided in the fragment data generation unit 22 of the fragmentation integration device 20 is a unit that can be processed when the processing device performs data processing based on the data size and the data pattern. Fragment data is generated by cutting out. For example, a pattern is searched forward (or backward) from the last point of the data read from a predetermined number of bytes. This pattern determines whether there is a punctuation character followed by a line feed code. When this pattern exists, the data up to that point is cut out as fragment data as a part that can be processed when the processing unit processes the data. This will be specifically described below.

  First, the fragment data generation unit 22 determines the type of data that the data reading unit 21 has started reading. Here, it is determined whether or not the data type is Japanese text data (step S201). If the data read by the data reading unit 21 is not Japanese text data (step S201 → No), the process is terminated. On the other hand, if the read data is Japanese text data (step S201 → Yes), the process proceeds to the next step S202.

  In step S202, the text fragment data generation unit 221 provided in the fragment data generation unit 22 first reads a predetermined amount of data. For example, data from the beginning of the Japanese text data shown in FIG. 4 to a point with a data size of 1024 KB is read.

  Next, the text fragment data generation unit 221 determines whether or not the end of the file exists when reading the data in step S202 (step S203). If the end of the file exists (step S203 → Yes), the text fragment data generation unit 221 cuts out as the last fragment data (step S207), and ends the fragment data generation processing. On the other hand, if the end of the file does not exist (step S203 → No), the process proceeds to the next step S204.

  In step S204, the text fragment data generation unit 221 performs a pattern search forward (or backward). Specifically, the text fragment data generation unit 221 determines whether or not there is a punctuation character followed by a line feed code (step S205). If there is no punctuation character and the following line feed code (step S205 → No), the pattern search continues further forward (or backward). On the other hand, when there is a punctuation character and a subsequent line feed code (step S205 → Yes), the text fragment data generation unit 221 cuts out to that point, generates fragment data (step S206), and proceeds to step S202. Return.

  The above process will be described using a specific example. As shown in FIG. 4, if the last point with a data size of 1024 KB from the beginning is the “flow” character at the point 401, the punctuation character “.” Followed by a new line The point where the code “↓” exists, that is, up to the end of the line of the line number “05” is generated as fragment data as a part that can be processed when the processing apparatus processes data. Next, in the second and subsequent times, reading is performed from the position immediately after cutting out as fragment data to the point of 1024 KB, not the point of 1024 KB from the beginning. In FIG. 4, assuming that a point with a data size of 1024 KB from the head of the line number “06” is a “chain” character at the point of reference numeral 402, a punctuation character “.” Followed by a new line The point where the code “↓” exists, that is, up to the end of the line of the line number “12”, is generated as fragment data that is a part that can be processed when the processing apparatus processes data.

  FIG. 6 is a diagram illustrating an example of fragment data generated by the text fragment data generation unit 221 of the fragmentation integration device 20 according to the first embodiment of the present invention. As shown in FIG. 6, in the fragment data generated by the text fragment data generation unit 221, the sentence is interrupted as Japanese by performing a pattern search as compared to the case where the data is simply divided based on the data size. Never. Therefore, the fragment data can be cut out as data as a unit that can be processed when the processing apparatus processes the data.

  It should be noted that the effect of shortening the response (response time) by the distributed processing using the distributed processing system 1 according to the first embodiment of the present invention can be expected in the following cases.

This is the case of D << P and M << P, where D is the data division time in the division integration device 20, M is the integration time in the division integration device 20, and P is the processing time in the processing device 40. For example, in the Japanese-English translation processing shown in the first embodiment of the present invention, the time for dividing Japanese text is D ₁ , the time for integrating English text is M ₁ , and the time for translation processing is P _1. Since the time D ₁ and the integration time M ₁ are much shorter than the translation processing time P ₁ , a significant distributed processing effect can be expected.

  Further, the text fragment data generation unit 221 according to the first embodiment of the present invention has been described as performing the fragment data generation processing of Japanese text data as the Japanese text fragment data generation unit. On the other hand, when the English text data is uploaded to the reception response device 10 and the processing device 40 performs an English-Japanese translation process and converts it into Japanese text data, the English text data is used as the text fragment data generation unit 221. A fragment data generation unit may be provided to perform fragment data generation processing of English text data. In this case, the English text fragment data generation unit reads a predetermined amount of data, and detects a period and a subsequent line feed code from the end of the read data forward or backward. Then, fragment data generation processing can be performed by cutting out the detected points as fragment data as a portion that can be processed when the processing device processes the data.

  As described above, according to the distributed processing system 1 according to the first embodiment of the present invention, the user terminal 2 can be obtained by preparing a plurality of processing devices 40 even for a single data having a large amount of information. The response speed can be increased in response to the processing request from. Therefore, the result can be returned to the user terminal 2 within the browser timeout time. In addition, it is possible to omit the construction of an accompanying system (system for e-mail address registration, e-mail transmission, etc.) such as notification of the processing result after the fact.

<< Second Embodiment >>
Next, a distributed processing system 1a according to a second embodiment of the present invention will be described. The difference from the distributed processing system 1 according to the first embodiment of the present invention is that the file processing unit 12 of the reception response device 10 further includes a sequential file processing unit 121 as shown in FIG. . The distributed processing system 1a according to the second embodiment, in addition to the effect of shortening the response (response time) by distributing and processing data obtained by the distributed processing system 1 according to the first embodiment, The sequential file processing unit 121 starts uploading data and activates the division integration device 20 to create a file while receiving data. And the process which transmits to the division | segmentation integration apparatus 20 in order from the file-ized data is performed, without waiting for upload of all the data. By doing so, the response (response time) is further shortened. This will be specifically described below.

  In the processing of the file processing unit 12 according to the first embodiment of the present invention, after the data reception unit 11 completes reception of all data, the data is filed and stored in the data storage unit 14, and then An activation instruction message was output to the dividing and integrating device 20 (see steps S102 to S104 in FIG. 2). On the other hand, when the data reception unit 11 starts to accept data from the user terminal 2, the sequential file processing unit 121 according to the second embodiment of the present invention first transmits an activation instruction message to the division integration device 20. Each time the sequential file processing unit 121 receives a predetermined amount (for example, 10 bytes) of data, the sequential data processing unit 121 converts the received data into a file and stores it in the data storage unit 14, and sequentially stores the stored data to the dividing and integrating device 20. Send.

  By doing in this way, without waiting for the upload of all the data by the reception response apparatus 10, it can transmit to the division | segmentation integration apparatus 20 in order from the received data, and a response (response time) can further be shortened. .

  FIG. 7 is a sequence diagram showing the flow of distributed processing of the distributed processing system 1a according to the second embodiment of the present invention. The same processes as those in the sequence according to the first embodiment of the present invention (see FIG. 2) are denoted by the same reference numerals and description thereof is omitted.

  First, when data is transmitted from the user terminal 2 via the network 3 (step S101), the data reception unit 11 of the reception response device 10 receives the data, and the sequential file processing unit 121 is triggered by the reception of the data. However, an activation instruction message is transmitted to the division integration device 20 via the transmission control unit 13 (step S301). And the division | segmentation integration apparatus 20 will start and will be in a data reading waiting state by receiving this starting instruction | indication message (step S302).

  Next, every time the data reception unit 11 receives a predetermined amount (for example, 10 bytes) of data, the sequential file processing unit 121 of the reception response device 10 sequentially converts the received data into a file and stores it in the data storage unit 14. The stored data is sequentially transmitted to the dividing and integrating device 20 (step S303). Then, the sequential file processing unit 121 determines whether reading of all uploaded data and transmission to the dividing and integrating device 20 have been completed (step S304). If there is data that has not yet been read and transmitted (step S304 → No), the process returns to step S303 to continue the processing. On the other hand, if all the data has been read and transmitted (step S304 → Yes), the process ends.

The data reading unit 21 of the dividing and integrating device 20 reads the data acquired from the reception response device 10 without waiting for the completion of reception of all data (step S105), and the read data is sent to the fragment data generating unit 22. hand over. Next, the fragment data generation unit 22 sequentially reads data and performs fragment data generation processing (step S106). Then, the data management unit 23 transmits the generated fragment data to the queue management device 30 (step S107).
In the following, the same processing as steps S108 to S120 shown in FIG. 1 is performed, the fragmentation result data is integrated in the division integration device 20, and a processing completion notification is transmitted to the user terminal 2.

  As described above, according to the distributed processing system 1a according to the second embodiment of the present invention, data upload processing from the user terminal 2 to the reception response device 10, fragment data generation processing by the division integration device 20, and Since the fragment data can be transmitted to the queue management device 30 in parallel, the response (response time) can be further shortened.

<< Third Embodiment >>
Next, a distributed processing system 1b according to a third embodiment of the present invention will be described. The difference from the distributed processing system 1 according to the first embodiment of the present invention is that data uploaded from the user terminal 2 is video data (video content). This video data is, for example, mpeg2 or H.264. H.264 video or its successor codec or derivative codec. In order to divide the video data, the fragment data generation unit 22 includes a video fragment data generation unit 222 in the fragment data generation unit 22 of the division integration device 20 as shown in FIG. Note that the processing performed by the processing device 40 for uploading the video data includes, for example, a cut point detection process for detecting a cut point at which one video is switched to another video, and a subject such as a person is greatly captured. For example, a noticed scene detection process for detecting a noticed scene such as an up-shot is provided.

This video fragment data generation unit 222 uses GOP (Group Of Pictures) as one non-dividable unit when cutting out fragment data from video data, and converts video data including a predetermined number of GOPs into one or more video data. Cut out as fragment data that is a set of video scenes.
Then, the fragment data generated by the video fragment data generation unit 222 is transmitted to the queue management device 30 by the data management unit 23 as in the first embodiment, and the processing device 40 performs the cut inspection for each piece of fragment data. Out processing and attention scene detection processing are performed.

Next, fragment data generation processing performed by the video fragment data generation unit 222 will be described.
FIG. 8 is a flowchart showing a flow of fragment data generation processing performed by the video fragment data generation unit 222 of the fragmentation integration device 20 according to the third embodiment of the present invention.

  First, the fragment data generation unit 22 determines the type of data that the data reading unit 21 has started reading. Here, the data type is mpeg2 or H.264. It is determined whether the video data is H.264 video or video data of a successor codec or a derived codec (step S401). If the data read by the data reading unit 21 is not video data such as mpeg2 (step S401 → No), the process ends. On the other hand, if the read data is video data such as mpeg2 (step S401 → Yes), the process proceeds to the next step S402.

  In step S402, the video fragment data generation unit 222 provided in the fragment data generation unit 22 cuts out data included in a predetermined number of GOPs. Next, the video fragment data generation unit 222 determines whether or not the end of the file exists when reading the data (step S403). When the end of the file exists (step S403 → Yes), the video fragment data generation unit 222 cuts out the read data as the last fragment data (step S405), and includes the fragment in the container (moving image file). Generate data. On the other hand, when the end of the file does not exist (step S403 → No), the video fragment data generation unit 222 includes the data extracted in step S402 in the container to generate fragment data (step S404). Return to step S402.

  Compared with the case where the processing for cutting out fragment data by executing decoding for each video is performed, according to the clipping in units of GOP by the video fragment data generation unit 222 according to the third embodiment of the present invention, the processing device 40 Compared to the processing time P, the division time D in the division integration device 20 can be greatly shortened, so that D << P can be realized. In addition, since the cut point detection process and the target scene detection process result in a frame number in the video, it is only necessary to generate an image corresponding to the frame number, and a series of uploaded video data data. Compared with the amount, the data amount of the output information is remarkably reduced, and the integration time M << processing time P in the dividing and integrating device 20 is satisfied, and the distributed processing effect according to the present invention can be enjoyed.

  Also, for example, the video of the mpeg2 codec is The present invention can also be applied to the case where the processing device 40 performs the process of converting to an H.264 codec video. However, when a plurality of converted fragment result videos (videos encoded in H.264) are combined, there is a case where the cost of the integration time M is high and the state of M << P cannot be realized. In that case, by integrating a plurality of video containers and using a playlist or the like that is regarded as one video container as a combined result, the load of integration can be reduced, and the state of M << P can be realized. It is possible to enjoy the distributed processing effect.

  As described above, according to the distributed processing system 1b according to the third embodiment of the present invention, a plurality of processing devices 40 are prepared when a single piece of video data (or content) with a large amount of information is input. By doing so, it becomes possible to shorten the response (response time) to the processing request from the user.

  In the distributed processing system 1b according to the third embodiment of the present invention, the fragment data generation unit 22 of the fragmentation integration device 20 includes the video fragment data generation unit 222. A fragment data generation unit may be provided. In this case, the audio fragment data generation unit processes the audio data received by the reception response device 10 by, for example, cutting out the data by separating a portion where the silent time of the audio data continues for a predetermined time as a delimiter. In this case, fragment data is generated as a unit that can be processed. Then, for example, data processing for converting the fragment data (voice data) into Japanese text data can be executed in the processing device 40 via the queue management device 30.

<< Modification of this embodiment >>
Next, a modified example of the distributed processing system 1 according to the embodiment of the present invention will be described. FIG. 9 is a configuration diagram for explaining a modification of the distributed processing system 1 according to the present embodiment. A system 100 shown in FIG. 9 is obtained by applying a postscript reference type DMS (Data Management System) 4 described in Patent Document 1 described above, instead of the reception response device 10 of the distributed processing system 1 shown in FIG. .

  In the DMS 4, when a plurality of data is uploaded from a plurality of user terminals 2 via the network 3, the reception distribution device 50 distributes and stores the received data to the plurality of additional recording devices 60, and performs a plurality of processes. The device 70 acquires data from the appending device 60 and processes the data. Then, the processing result of each processing device 70 is transmitted to the reception distribution device 50 and returned to the user terminal 2. Thus, according to the DMS 4, a plurality of data can be processed in parallel, and the throughput of the entire system can be improved.

  In this system 100, the DMS 4 is provided instead of the reception response device 10 of the distributed processing system 1 according to the present embodiment. Here, it is assumed that the processing device 70 of the DMS 4 performs the Folder Engine processing. For example, when the user selects a plurality of data files using the GUI function of the user terminal 2 and batch DROPs the folder, the Folder Engine process is a process indicated by the folder name as a processed folder. , And each processing result is stored in a folder. In this Folder Engine processing, a folder with processing can be further inserted into one folder with processing by the nested structure of folders with processing. For example, there is a case in which certain image data is converted into Japanese text data by OCR processing, further, Japanese-English translation processing is performed on the Japanese text data, and the English summarization processing is further performed. Such processing can be collectively processed by the nested structure of folders with processing.

  When this processing is performed in the present system 100, data is acquired from the processing device 70 of the DMS4. (1) First, the OCR processing device 45 performs OCR processing. (2) Next, the result data is The processing device 70 transmits to the division integration device 20 of the distributed processing system 1. Then, the data is divided by the division integration device 20 to generate fragment data. Subsequently, each processing device 40 acquires the fragment data stored in the queue management device 30, translates the Japanese text data into English text data, and transmits the fragment result data to the fragmentation integration device 20. The fragmentation integration device 20 integrates the fragment result data, and transmits the integrated result data to the processing device 70 of the DMS 4. (3) Then, the English summary processing unit 46 performs English text data summarization processing. By performing such processing, a response (response time) by the processing device 70 can be shortened. Further, when these processes are requested simultaneously from a large number of user terminals 2 (that is, when uploading at the same time), the processing device 70 operates simultaneously, thereby improving the system throughput.

  As described above, according to the present system 100, it is possible to improve the throughput with respect to requests from a large number of users by using the DMS4. Further, even for data having a large amount of information input from the users, By combining the distributed processing system 1 according to the present embodiment, the response (response time) can be improved.

1 distributed processing system 2 user terminal 3 network 5 DMS
DESCRIPTION OF SYMBOLS 10 Reception response apparatus 11 Data reception part 12 File processing part 13 Transmission control part 14 Data accumulation part 20 Fragmentation integration apparatus 21 Data reading part 22 Fragment data generation part 23 Data management part 24 Fragment result data acquisition part 25 Fragment result data integration part 26 Result data output unit 27 Data storage unit 30 Queue management device 31 Queue reception unit 32 Queue management unit 33 Queue storage unit 40, 70 Processing device 41 Queue selection unit 42 Fragment data acquisition unit 43 Data processing unit 44 Fragment result data output unit 45 OCR Device 46 English summary processing device 50 Acceptance distribution device 60 Additional recording device 121 Sequential file processing unit 221 Text fragment data generation unit 222 Video fragment data generation unit

Claims (7)

(1) an acceptance response device that accepts upload data from a user terminal via a communication network, and returns the result data of data processing on the accepted upload data to the user terminal synchronously or asynchronously; (2) the acceptance response Dividing the upload data acquired from the apparatus to generate a plurality of fragment data, and integrating the plurality of fragment result data, which is a result of data processing of the generated plurality of fragment data, to generate the result data An integration device; (3) one or more pieces of data obtained from the fragmentation integration device, stored as a queue, and sent to the processing device in response to a request from one of a plurality of processing devices; A queue management device; (4) acquiring the fragment data from the queue management device; The by data processing, said plurality of processing units to be transmitted to the cutting integrated device as said fragments result data, a distributed processing system comprising,
The reception response device includes:
A data reception unit that receives the upload data from the user terminal via the communication network, and obtains the upload data from the data reception unit into a file, and stores the filed data in a data storage unit. A file processing unit that generates a start instruction message for starting the division integration device, a start instruction message generated by the file processing unit and the filed data are transmitted to the division integration device, and acquired from the division integration device A transmission control unit that transmits the result data to the user terminal via the communication network;
The dividing and integrating device is
A data reading unit for reading the data transmitted from the acceptance response device in order from the beginning of the data, a type of data read by the data reading unit, and the processing device according to the determined type of data Generates a plurality of fragment data by cutting out in units that can be processed when the data is processed, and the fragment data generation unit generates one of the plurality of fragment data. A data management unit that sequentially assigns each of the plurality of pieces of fragment data and transmits the fragment data to the one or more queue management devices;
Receiving the fragment result data transmitted from the plurality of processing devices, and all the fragment result data based on sequential IDs attached to the fragment result data in association with the IDs attached to the fragment data When the fragment result data acquisition unit that determines whether or not the fragment result data acquisition unit has acquired all the fragment result data, the sequential result attached to the fragment result data is determined. Using the ID, the fragment result data integration unit that integrates the fragment result data in the ID order and generates the result data, the data storage unit that stores the result data generated by the fragment result data integration unit, A result data output unit that transmits the result data stored in the data storage unit to the acceptance response device;
The queue management device includes:
The queue accepting unit that accepts the fragment data transmitted from the fragmentation integration device and saves the fragment data as a queue in a queue storage unit, and the fragment data acquisition request from the processing device, the queue storage unit stores the fragment data A queue management unit that transmits fragment data to the processing device, and the queue storage unit that stores the fragment data received by the queue reception unit,
The processor is
The fragment data acquisition request for transmitting the fragment data acquisition request to the queue management device and acquiring the fragment data from the queue management device; and predetermined data processing for the fragment data acquired by the fragment data acquisition unit A data processing unit to be executed; and the fragmentation integration device that attaches a sequential ID associated with the ID assigned to the fragment data to the fragment result data that is a result of data processing by the data processing unit Fragment result data output section to be transmitted to,
A distributed processing system comprising: The reception response device includes:
When the data accepting unit starts accepting the upload data, the start instruction message is generated and transmitted to the dividing and integrating device via the transmission control unit, and the upload data received by the data accepting unit is transmitted. Each time a predetermined amount is acquired, the data is divided into files, the filed data is sequentially stored in the data storage unit, and each time the data is stored in the data storage unit, the data is transmitted to the transmission control unit. Further comprising a sequential file processing unit that transmits to the dividing and integrating device via
The dividing and integrating device is
2. The distribution according to claim 1, wherein each time the data reading unit reads the data from the reception response device, the data reading unit transfers the data to the fragment data generation unit, and the fragment data generation unit generates the fragment data. Processing system. The fragment data generation unit of the fragmentation integration apparatus is
When the type of the data read by the data reading unit is Japanese text data, the data is read up to a predetermined amount, and a punctuation character and subsequent line feed code are detected forward or backward from the end of the read data. And a Japanese text fragment data generation unit that extracts up to the detected point as fragment data as a processable part when the processing device processes the data,
The distributed processing system according to claim 1, further comprising: The fragment data generation unit of the fragmentation integration apparatus is
When the type of the data read by the data reading unit is English text data, the data is read up to a predetermined amount, and a period and a subsequent line feed code are detected forward or backward from the end of the read data, An English text fragment data generation unit that extracts up to the detected point as fragment data as a portion that can be processed when the processing device processes the data.
The distributed processing system according to claim 1, further comprising: The fragment data generation unit of the fragmentation integration apparatus is
Video fragment data generation for generating fragment data by cutting out the video data for each predetermined number of GOPs when the type of the data read by the data reading unit is GOP (Group Of Pictures) units Part
The distributed processing system according to claim 1, further comprising: (1) an acceptance response device that accepts upload data from a user terminal via a communication network, and returns the result data of data processing on the accepted upload data to the user terminal synchronously or asynchronously; (2) the acceptance response Dividing the upload data acquired from the apparatus to generate a plurality of fragment data, and integrating the plurality of fragment result data, which is a result of data processing of the generated plurality of fragment data, to generate the result data An integration device; (3) one or more pieces of data obtained from the fragmentation integration device, stored as a queue, and sent to the processing device in response to a request from one of a plurality of processing devices; A queue management device; (4) acquiring the fragment data from the queue management device; The by data processing, said plurality of processing units to be transmitted to the cutting integrated device as said fragments result data, a distributed processing method for use in a distributed processing system comprising,
The reception response device includes:
A data storage unit for storing the uploaded data as a file;
Receiving the upload data from the user terminal via the communication network; obtaining the received upload data into a file; and storing the filed data in the data storage unit; Performing a step of generating a start instruction message for starting the process, and a step of transmitting the generated start instruction message and the filed data to the dividing and integrating device,
The dividing and integrating device is
A step of reading the data transmitted from the acceptance response device in order from the beginning of the data, a type of the read data is determined, and the processing device processes the data according to the determined type of data Generating a plurality of fragment data by cutting out in units that can be processed, and assigning a sequential ID to each of the plurality of fragment data in the order in which one of the plurality of fragment data is generated. And transmitting to the one or more queue management devices,
The queue management device includes:
A queue storage unit that stores the fragment data transmitted from the fragmentation integration device as a queue;
Receiving the fragment data transmitted from the fragmentation integration device and storing the fragment data in a queue storage unit as a queue; and the fragment data stored in the queue storage unit based on an acquisition request for the fragment data from the processing device Transmitting to the processing device, and
The processor is
Transmitting the fragment data acquisition request to the queue management device, acquiring the fragment data from the queue management device, executing predetermined data processing on the acquired fragment data, and the data processing Performing a step of attaching a sequential ID associated with the ID attached to the fragment data to the fragment result data which is a result obtained, and transmitting the fragment ID to the fragmentation integration device,
The dividing and integrating device is
A data storage unit for storing the result data obtained by integrating the fragment result data;
The fragment result data transmitted from the plurality of processing devices is received, and all the fragment results are based on the sequential ID attached to the fragment result data in association with the ID attached to the fragment data. A step of determining whether or not data has been acquired; and when it is determined that all of the fragment result data has been acquired, the sequential IDs attached to the fragment result data are used to determine the fragments in the order of the IDs. Integrating the result data and generating the result data; storing the result data generated by the fragment result data integration unit in the data storage unit; and storing the result data stored in the data storage unit Transmitting to the reception response device,
The reception response device includes:
The distributed processing method characterized by performing the step which transmits the said result data acquired from the said division | segmentation integration apparatus to the said user terminal via the said communication network.   A program for causing a computer to execute the distributed processing method according to claim 6.

Images