JP2014229969A - Parallel distributed video analysis device, parallel distributed video analysis method, and parallel distributed video analysis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the utilization rate of hardware resources.SOLUTION: Analysis processing of video data is started by unused processes P1, P2 assigned to a task. If the assignment number of processes for that task is insufficient, the processing section of not yet analyzed video data is prorated and assigned to four processes P1-P4, respectively, after the processes P3, P4, assigned to other task, are not used.

Description

  The present invention relates to a technique for analyzing video.

  Image analysis technology that detects events such as cut points, camera work, telops, music sections, etc., using physical characteristics (pixel brightness, audio frequency, audio intensity, etc.) of image and sound data extracted from frames in the video Yes (Patent Documents 1 and 2).

  In this technique, since an arithmetic process for sequentially comparing and subtracting two or more frames that are temporally related in sequence is performed from the beginning to the end of the video, it takes time proportional to the number of frames.

  Therefore, in Patent Document 3, the video data is divided into a plurality of processing sections, and the adjacent processing sections are overlapped with each other and analyzed in parallel, thereby accurately detecting the event occurrence section. , Video analysis processing time has been shortened. Further, in Non-Patent Document 1, an event occurrence interval is detected in a minimum overlap processing interval by dynamically extending the overlap interval according to the event detection state.

Japanese Patent No. 2839132 Japanese Patent No. 3488626 Japanese Patent No. 3785068

Goto and 2 others, “Acceleration of video event detection by parallel distribution”, Information Processing Society of Japan, Vol.2012-AVM-76, No.2, February 23, 2012, p.1-6

  However, since there is an upper limit to the number of processes that can be executed simultaneously and in parallel due to physical constraints of hardware resources, there are cases where processes cannot be allocated to all divided processing sections.

  In particular, when there are a plurality of video data, the total number of divided processing sections becomes enormous, and therefore it is further difficult to always assign processes to all the processing sections.

  That is, there is a problem that a sufficient number of processes cannot be secured for a task for analyzing video data. In this regard, the process allocation process waits until a sufficient number of processes are secured, and when the number of processes that can be allocated can be secured when other tasks are completed, the process of video analysis is started. There is also a way to do it. However, the waiting time until the task is started becomes long, and the operation rate of the hardware resource is lowered.

  The present invention has been made in view of the above circumstances, and an object thereof is to improve the operating rate of hardware resources.

  The parallel distributed video analysis apparatus according to claim 1, an analysis unit that starts an analysis process of video data read from a storage unit by an unused first process assigned to a task for video analysis, and the task When the number of processes allocated to the number is insufficient, after the second process allocated to another task becomes unused, the unprocessed processing section of the video data is apportioned, and the first process and the The gist of the present invention is to have allocation means for allocating to each of the second processes.

  The parallel and distributed video analysis device according to claim 2 is the parallel and distributed video analysis device according to claim 1, wherein the assigning unit is arranged between the analyzed processing sections among the plurality of unprocessed sections that are prorated. And assigning an unprocessed section having continuity to the first process.

  The parallel distributed video analysis method according to claim 3 includes an analysis step of starting analysis processing of video data read from the storage means by a first unused process assigned to a video analysis task by a computer. When the number of processes allocated to the task is insufficient, after the second process allocated to another task becomes unused, the unprocessed processing section of the video data is apportioned, and the first process The gist of the present invention is to have a process and an assigning step to allocate to each of the second processes.

  The parallel and distributed video analysis method according to claim 4 is the parallel and distributed video analysis method according to claim 3, wherein the assigning step is performed between an analyzed processing section and a plurality of allocated unprocessed sections. And assigning an unprocessed section having continuity to the first process.

  The gist of the parallel distributed video analysis program according to claim 5 is to cause a computer to execute the parallel distributed video analysis method according to claim 3 or 4.

  As described above, according to the present invention, when the video data analysis processing is started by the unused first process assigned to the task and the number of processes assigned to the task is insufficient, the process is assigned to another task. After the used second process becomes unused, the unprocessed processing section of the video data is apportioned and allocated to the first process and the second process, respectively. Can be kept high.

  Further, according to the present invention, an unprocessed section having continuity with an analyzed processing section among a plurality of unprocessed sections thus allocated is allocated to the first process. Generation of resource management for generation and seek from the top of the video to the processing start frame) can be avoided, and the operating rate of hardware resources can be kept higher.

  According to the present invention, the operating rate of hardware resources can be kept high.

It is a figure which shows the example of the task scheduling at the time of process addition reallocation. It is a figure which shows the functional block structure of a parallel distributed video analyzer. It is a flowchart which shows the process from task reception to task start. It is a figure which shows the example of task data. It is a flowchart which shows the process after completion of a task. It is a flowchart which shows the whole image | video analysis process. It is a figure which shows the example of worker state data. It is a flowchart which shows a new allocation process. It is a flowchart which shows an additional reallocation process. It is a figure which shows the example of a new allocation process result. It is a figure which shows the example of an additional reallocation process result. It is a flowchart which shows an event detection process. It is a figure which shows the example of a process area extension result. It is a flowchart which shows a detection result integration process. It is a figure which shows the construction example of a parallel distributed image | video analysis apparatus.

  First, the present invention will be outlined.

  In the present invention, when the number of processes required at the time of video analysis cannot be sufficiently secured, tasks are started within the range of the number of processes that can be allocated, and processes can be allocated as other tasks are completed. By allocating unprocessed sections of tasks that have not been sufficiently secured at timing according to the process being processed and the process to compensate for the shortage, the operation rate of hardware resources is kept high.

  Also, when allocating, considering the continuity with the processing section being analyzed by the process being processed, resource management for generating the process, seeking from the beginning of the video to the processing start frame, etc. The generation of overhead is avoided.

  This will be described with reference to FIG. For example, it is assumed that four processes P1 to P4 are designated for the task T1 for analyzing a certain video data V1. Here, even if only two processes P1 and P2 are actually assigned, the analysis process of the video data V1 is performed by the two processes P1 and P2 (video analysis is performed on a total of two divided processing sections in parallel). Is started (FIG. 1A).

  Then, after the analysis processing in the other task T2 is completed and the processes P3 and P4 can be used, the unprocessed sections R1 and R2 of the processes P1 and P2 are divided into four processes in the task T1 still being processed. Apportioning is performed at P1 to P4 (FIG. 1B).

  Thereafter, the four unprocessed sections R11, R12, R21, and R22 that are prorated are assigned to the processes P1 to P4, respectively, and the analysis process (a total of four prorated processes are allocated in parallel) by the four processes P1 to P4. The video analysis is continued (FIG. 1 (c)). At this time, unprocessed sections R11 and R21 that are continuous with the process sections so far are assigned to the processes P1 and P2 that are being processed.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. Hereinafter, an environment for executing a process is called a worker. For example, a CPU or multi-core CPU has a physical core, and a CPU capable of hyperthreading has a logical core. For processing that depends on CPU processing performance such as video processing, it is inefficient to execute a plurality of processes with one worker, and it is preferable to allocate one process to one worker for execution.

  FIG. 2 is a diagram showing a functional block configuration of the parallel distributed video analysis apparatus 1 according to the present embodiment. The parallel distributed video analysis apparatus 1 includes a task reception unit 11 that receives input of a task for video analysis, a task storage unit 12 that stores the task data, a video storage unit 13 that stores input video data, A task assignment unit 14 that executes worker assignment processing to tasks, a worker state storage unit 15 that stores worker state data indicating the processing state of the worker, a worker unit 16 that analyzes video data by a plurality of workers, The intermediate result storage unit 17 stores each analysis result as intermediate result data, the result integration unit 18 that integrates the intermediate result data, and the processing result storage unit 19 that stores the integrated data.

  Each of these functional units can be realized by a computer having a memory and a CPU. Further, the processing of each functional unit can be executed by a program. These are connected to each other so as to be able to communicate with each other using a communication means such as a wired connection using a bus, an Ethernet cable, a USB cable, or a wireless LAN.

  Next, the operation of the parallel distributed video analyzer 1 will be described.

  First, the characteristic operation described above will be mainly described with reference to FIGS.

  FIG. 3 is a flowchart showing processing from task reception to task start. First, the task reception unit 11 receives input of a video analysis task generated by a user and video data analyzed by the task, and stores them in the task storage unit 12 and the video storage unit 13 (step S101). ).

  An example of task data is shown in FIG. The task data includes the task ID, the video data ID analyzed by the task, the number of workers specified by the task, the worker ID assigned to the task, and the processing of the task. A processing state flag indicating the state is stored in association with each other.

  For example, when a video data ID and a specified number of workers are input by a user, one task with a task ID for that is generated and added to the task data record. However, the processing status flag is set according to the processing stage among “Before processing”, “Processing”, and “Processing completed”, and the assigned worker ID is input when the task is processed.

  Next, the task assignment unit 14 acquires task data from the task storage unit 12 and determines whether there is a “before processing” or “in processing” task other than the input task (step S102).

  If there is no task having these processing status flags, the task assignment unit 14 assigns a worker that can be used to the input task, and the worker unit 16 immediately applies only to the assigned worker. The analysis processing of the video data to be started is started (step S103).

  On the other hand, if there is a task having these processing status flags, the task allocation unit 14 sets the processing status flag of the input task to “before processing” (step S104), and the task with the oldest input is set. Then, a worker that can be used at present is allocated, and analysis processing of the corresponding video data is started only by the worker (step S105).

  FIG. 5 is a flowchart showing processing after the task is completed. First, the task assignment unit 14 acquires task data from the task storage unit 12, and sets the processing status flag of the task for which analysis processing has been completed to “processing completed” (step S201).

  Next, the task allocating unit 14 extracts, from the task data, a task whose processing state flag is “processing” and corresponds to “specified worker count> allocated worker count” (step S202).

  If such a task exists, the task allocation unit 14 compensates for the shortage of workers using a worker that is newly available upon completion of the processing of the task, and increases by adding the specified number of workers and supplementation. The unprocessed section is apportioned and reassigned with the minimum value of the number of workers, and the worker unit 16 resumes the analysis processing of the unprocessed section (step S203).

  This is the end of the description of the characteristic operation.

  Next, the overall operation of the video analysis processing performed by the parallel distributed video analysis device 1 will be described with reference to FIGS.

  FIG. 6 is a flowchart showing the entire video analysis process. First, with the input of a task, the task assignment unit 14 acquires all the tasks from the task storage unit 12 (step S301), and confirms the current processing state in the input order of each task (step S302).

  Next, the task assignment unit 14 determines whether there is a “processing” task (step S303). If there is a task, the task assignment unit 14 determines whether there is a task whose assigned worker number is smaller than the specified worker number. Further determination is made (step S304), and if it does not exist or if the determination result in step S304 is No, it is further determined whether or not a “pre-processing” task exists (step S305).

  If the determination result in step S304 is Yes, the worker reassignment process is executed as described with reference to FIG. 5 (step S306). If the determination result in step S305 is Yes, A new worker assignment process is executed (step S307).

  Thereafter, the worker unit 16 resumes or starts the analysis processing in parallel by the plurality of assigned workers (step S308), and the result integration unit 18 integrates each analysis result (step S309).

  If the processing state of all tasks is “processing complete”, the determination result in step S304 is No, and this flow is immediately terminated.

  Here, worker status data stored in the worker status storage unit 15 will be described. FIG. 7 is a diagram illustrating an example of worker status data. The worker status data includes the worker ID, the processing section ID of the video data assigned to the worker, the time stamp of the processing frame, the time stamp of the processing start frame, and the time stamp of the processing end frame. An operation state flag indicating the operation state of the worker is stored in association with it.

  In the operation state flag, “processing” or “stopped” is set according to the operation state. “Processing” indicates that a processing section has already been assigned and analysis processing is being executed. “Stopped” means that a processing section is not allocated and a new processing section can be allocated.

  Further, the time stamp of the process start frame and the time stamp of the process end frame represent the start point and end point of the assigned processing section. The worker executes event detection processing on the frames included in this processing section. The processing frame represents a time stamp of a frame that is currently being processed by the worker.

  Next, a new allocation process (step S307 in FIG. 6) for assigning a worker to the “before process” task will be described in detail with reference to FIG.

  First, the task assignment unit 14 acquires worker IDs whose operation state flag is “stopped” from the worker state storage unit 15 and calculates the total number a (step S401).

  Next, the video data corresponding to the “pre-processing” target task is acquired from the video storage unit 13, and the video length L is calculated (step S402).

  Next, the minimum value between the designated worker number r designated for the target task and the allocatable worker number a calculated in step S401 is set as the division number d (step S403).

Next, the processing start frame s _i and the processing end frame e _i of the processing section P _i (1 ≦ i ≦ d) are calculated from the following formulas (1) and (2) (step S404). The processing section P _i (s _i , e _i ) is assigned to the acquired “stopped” worker W _n (step S405).

Next, the worker _{W n} the operation status flag is changed to "process" (step S406), increments the value of i (step S407), "i ≧ d + 1" step S404~ step S407 until is satisfied Is repeated (step S408).

  After that, when the processing section new allocation processing is completed for all workers, the determination result in step S408 is Yes, and the processing status flag of the target task that was “Before processing” is changed to “Processing” (Step S408). S409), this flow ends.

  Next, an additional reassignment process (step S306 in FIG. 6) for adding a worker to the task ““ processing ”and worker shortage” will be described in detail with reference to FIG.

  First, the task assignment unit 14 acquires worker IDs whose operation state flag is “stopped” from the worker state storage unit 15 and calculates the total number a (step S501).

  Next, the “processing in progress” worker ID already assigned to the target task “in processing” and “worker shortage” is acquired from the task storage unit 12 and the worker state storage unit 15, and the total number p is obtained. Calculate (step S502).

Next, the processing end frame e _j and the processing frame c _j corresponding to the “processing” worker ID acquired in step S502 are acquired from the worker state storage unit 15 (step S503).

Next, from the following equation (3), the length of each unprocessed section for which analysis processing has not been executed in the “processing” worker is calculated, and the total unprocessed section length Lr of the target task is calculated. (Steps S504 to S505).

  Next, the designated worker number r designated for the target task is compared with the total value of the assignable worker number a calculated in steps S501 and S502, respectively, and the “pending” worker number p. Then, the minimum value is set as the division number d (step S506).

Next, a processing start frame s _i and a processing end frame e _i of a new processing section P _i (1 ≦ i ≦ d) are calculated from the following equations (4) and (5) (step S507).

  Next, it is determined whether or not there is a “processing” worker in the target task (step S508). If there is a worker, the processing task already assigned to the “processing” worker is determined. New processing sections are continuously allocated so that processing can be continued (step S509). As a result, it is possible to avoid the occurrence of overhead such as useless seek, which occurs due to discontinuity of resource management and processing sections related to “processing” workers.

  On the other hand, if there is no “processing” worker, a processing section is arbitrarily assigned to the “stopping” worker (step S510), and the operating state flag of the worker is changed to “processing” (step S511). .

  Then, after step S509 or step S511, the value of i is incremented (step S512), and steps S507 to S512 are repeated until “i ≧ d + 1” is satisfied (step S513).

  After that, when the processing section additional reassignment processing is completed for all workers, the determination result in step S513 is Yes, and the processing status flag of the task that was “processing” is changed to “processing”. The flow is terminated (step S514).

An example of a new allocation result and an example of an additional reassignment result are shown in FIGS. 10 and 11, respectively. In the case of new allocation, as shown in FIG. 10, the processing section P ₁ [0 to 1,000 (ms)] is allocated to the worker W ₁ and the processing section P ₂ [1,000 to 2,000 (ms)]. There is assigned to the worker _{W 2,} the processing block _P 3 _[2,000~3,000 (ms)] is assigned to the worker _{W 3,} the processing section _P 4 _[3,000~4,000 (ms)] is worker assigned to W _4, the processing section _P 5 _[4,000~5,000 (ms)] is assigned to the worker _{W 5,} detection processing of the event is executed in parallel.

FIG. 11 shows a specific example in which workers W ₃ and W ₄ are additionally reassigned to tasks being processed by workers W ₁ and W ₂ . The worker W ₁ is assigned a processing section P ₁ [0 to 5,000 (ms)], and the worker W ₂ is assigned a processing section P ₂ [5,000 to 10,000 (ms)]. The unprocessed section lengths of the respective processing sections P ₁ and P ₂ are both 4,500 (ms).

In this case, since the processing section of 9,000 (ms) is unprocessed in the entire video, the total unprocessed section length is divided into four sections and assigned to each worker. In this example, the newly calculated processing section P ₁ [500-2,750 (ms)] is assigned to the worker W ₁ and the processing section P ₂ [2,750-5,000 (ms)] is assigned to the worker. Assigned to W ₂ , processing section P ₃ [5,500 to 7,750 (ms)] is assigned to worker W ₃ , and processing section P ₄ [7,750 to 10,000 (ms)] is assigned to worker W _4. Assigned to.

  When calculating the processing section, in addition to the method of equally dividing the number of workers to which the unprocessed section is allocated, considering the processing speed of the worker and the length of the unprocessed section, the processing completion time of all workers is made equal. A prorated method is possible.

  Next, the event detection process (step S308 in FIG. 6) will be described in detail with reference to FIG.

  First, the worker unit 16 starts event detection processing from the processing start frame of the processing section assigned by the task assignment unit 14 by a plurality of workers (step S601), and at the same time, the worker state storage unit 15 operates in the operation state. Is notified as “processing” (step S602).

  When the process end frame is reached (step S603), the event detection state (“detecting” or “not detected”) in the frame being processed is determined (step S604), and the processing interval is determined based on the determination result. Judge whether to extend.

  Here, if the event is “detecting”, that is, if the event continues until the next processing interval, the event detection process is performed while extending the processing interval until the currently detected event ends. (Step S605).

  After that, when the event detection process is completed, the determination result in step S604 is No, the detected event occurrence section (start frame time stamp and end frame time stamp), and auxiliary information (for example, camera) The camera movement information at the time of workpiece detection and the appearance coordinates of the telop at the time of telop detection) are transmitted to the intermediate result storage unit 17 (step S606), and its own operation state in the worker state storage unit 15 is updated to “stopped”. (Step S607), and this flow ends.

An example of the processing section extension result is shown in FIG. For example, as shown in FIG. 13A, the processing section P ₁ [0 to 10,000 (ms)] is assigned to the worker W ₁ and the processing section P ₂ [10,000 to 20,000 (ms)]. there is assigned to the worker _{W 2,} when it reaches the time stamp of the processing end frame [10,000 (ms)], and was in the detection of an event in worker _{W 1.}

In this case, the worker W ₁ continues the analysis process while changing the time stamp of the process end frame to the rear of the video, and when the event detection process ends (here, the time is shown) as shown in FIG. The analysis process ends at the point where the stamp is [13,000 (ms)].

By extending the processing section in this way, the end point of the processing section P ₁ analyzed by the worker W ₁ is changed backward by 3,000 (ms), so that it is stored in the worker state storage unit 15. The data of the processing section to be updated is updated. That is, the data of the processing section in which the processing start point is 0 (ms) and the processing end point is 10,000 (ms) are changed to 0 (ms) and 13,000 (ms), respectively, and the worker state It is stored in the storage unit 15.

  Next, the detection result integration process (step S309 in FIG. 6) will be described in detail with reference to FIG.

  First, the result integration unit 18 acquires each detection result of the target task from the intermediate result storage unit 17 (step S701), and integrates them as one processing result in the order of processing intervals (step S702).

  At this time, if the processing section is extended in the analysis process, the overlapping video section is processed by a plurality of workers. Therefore, it is determined whether there is a detection result of the overlapping processing section (step S703). If there is a detection result, the detection result detected in the processing section temporally close to the head of the video is displayed. Integration is performed by regarding it as a correct detection result (step S704).

  Then, after integrating all the processing results detected in the target task (step S705), the processing status flag of the task is changed to “processing completed” (step S706). Thereafter, the task storage unit 12 is searched again, and steps S701 to S706 are repeated until there is no “pre-processing” task. If there is no “before processing” task, the process waits until a new task is input.

  This is the end of the description of the video analysis process.

  FIG. 15 shows a construction example of the parallel distributed video analysis apparatus 1 described so far. As shown in FIG. 15A, one core (1) of a multi-core CPU performs task management as a task allocation unit, and a plurality of cores (2) to (4) connected thereto serve as a plurality of workers W. The video analysis may be performed in parallel.

  Further, as shown in FIG. 15B, one server performs task management as a task allocation unit, and a plurality of servers (1) to (3) connected to the server perform video as a plurality of workers W in parallel. You may make it analyze.

  In addition to these, the video analysis processing may be executed in parallel by a plurality of servers having a multi-core CPU. In this way, any form of worker construction can be adopted as long as each worker can operate independently.

  As described above, according to the present embodiment, when the number of workers required at the time of video analysis cannot be sufficiently secured, the task is started within the range of the assignable number of workers, and the worker is completed when other tasks are completed. Since the unprocessed section of the task that could not be secured sufficiently is allocated with the worker that is processing and the worker that compensates for the shortage at the timing when the allocation becomes possible, while keeping the operating rate of hardware resources high Realize high-speed video analysis by parallel distribution.

  Further, according to the present embodiment, since the continuity with the processing section being analyzed by the worker being processed is taken into consideration at the time of allocation, the occurrence of overhead can be avoided and the operating rate of hardware resources can be reduced. It can be kept higher.

  In addition, according to the present embodiment, when an event is being detected at the end of the divided processing section, the processing section is extended to generate an overlapping section in the processing section with another worker. In addition to being able to perform analysis processing, it is possible to perform video analysis with more efficient operation without causing many unnecessary overlap sections.

  In addition, by installing a program for executing the function of each functional unit of the parallel distributed video analysis apparatus 1 described above in a computer, a video analysis processing program for causing the computer to function as the parallel distributed video analysis apparatus 1 may be constructed. Is possible.

DESCRIPTION OF SYMBOLS 1 ... Parallel distributed video analyzer 11 ... Task reception part 12 ... Task memory | storage part 13 ... Video | video memory | storage part 14 ... Task allocation part 15 ... Worker state memory | storage part 16 ... Worker part 17 ... Intermediate | middle result memory | storage part 18 ... Result integration part 19 ... Processing result storage unit S101 to S105, S201 to S203, S301 to S309, S401 to S409, S501 to S514, S601 to S607, S701 to S706 ... step

Claims (5)

Analysis means for starting analysis processing of video data read out from the storage means by an unused first process assigned to a task for video analysis;
When the number of processes allocated to the task is insufficient, after the second process assigned to another task becomes unused, the unprocessed processing section of the video data is apportioned, and the first process And assigning means for assigning to each of the second processes,
A parallel distributed video analysis apparatus characterized by comprising: The assigning means includes
2. The parallel distributed video analysis according to claim 1, wherein among the plurality of unprocessed sections that are prorated, an unprocessed section that is continuous with the analyzed processing section is assigned to the first process. apparatus. By computer
An analysis step of starting analysis processing of video data read from the storage means by an unused first process assigned to a task for video analysis;
When the number of processes allocated to the task is insufficient, after the second process assigned to another task becomes unused, the unprocessed processing section of the video data is apportioned, and the first process And assigning steps respectively assigned to said second processes;
A parallel distributed video analysis method characterized by comprising: The assigning step includes
4. The parallel distributed video analysis according to claim 3, wherein, among the plurality of apportioned unprocessed sections, an unprocessed section having continuity with the analyzed process section is allocated to the first process. 5. Method.   A parallel distributed video analysis program that causes a computer to execute the parallel distributed video analysis method according to claim 3.

Images