JP2003069946A - Video analyzer, video analysis method, video analysis program and its program recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a video analysis time without deteriorating detection accuracy in the video image analysis for analyzing scene information or the like. SOLUTION: In the video analyzer 1 for analyzing scene information or the like, a video input section 11 stores video data inputted from an external device and a video division section 12 divides the video data into partial video images of a proper number. The devided video data are distributed respectively to video analysis sections (1 to N) 13, which perform video image analysis in parallel. An analysis result integration section 14 receives partial analysis results from the video analysis sections (1 to N) 13 and integrates them to obtain a single analysis result. An analysis result output section 15 outputs the analysis result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video analysis technique for reading a video signal, switching scenes in video contents, analyzing scene information such as presence / absence of telop characters and presence / absence of music.

[0002]

2. Description of the Related Art Many image analysis techniques for analyzing scene information such as cuts, telops, and sounds in images have been proposed. (References) [1] Otsuji et al., "Moving Image Browsing Using Luminance Information", IEICE Technical Report, IE90-103,
1991. [2] Taniguchi et al., “Method and apparatus for detecting video cut points”,
Patent No. 2839132. [3] Sotomura et al., "Scene Extraction Processing Method", Patent No. 2
No. 960939 [4] Nakajima et al., “Creating a moving image summary by extracting keyword images”, Jikojo Zendai 1F-10, 1994 latter term.

In the conventional methods, in order to analyze the contents of a scene in a video, the comparison calculation processing of adjacent frame images is sequentially performed from the beginning to the end of the video.

As an example of comparison calculation processing of adjacent frame images, the cut detection method proposed in Reference [1] calculates the difference in luminance value between corresponding pixels of adjacent frame images. When the sum of the absolute values over the entire screen is larger than a predetermined threshold value, it is detected as a cut.

[0005]

In the prior art, when analyzing the entire video, the comparison calculation processing of two adjacent frame images is sequentially performed from the beginning to the end of the video, so the processing required for the analysis is performed. The time required was proportional to the number of frame images that compose the video. Therefore, there is a problem that the analysis result cannot be obtained in a shorter time.

In order to solve this problem, as a conventionally proposed method, there is a method of performing comparison processing every few frames instead of comparing adjacent frame images. However, when the frame image interval for comparison is increased in this way, the scene in which the person or object moves in the same scene may be erroneously detected as a scene change, and the detection accuracy decreases. There was also a problem.

An object of the present invention is to analyze a scene information such as a cut, a telop, a sound in a video in a video analysis,
It is possible to reduce the video analysis time as compared with the conventional method without lowering the detection accuracy.

[0008]

In order to solve the above problems, the present invention temporarily divides input video data, which is the target of video analysis, into partial videos, and processes the video analysis of these partial videos. Is performed in parallel, the image analysis results generated by the image analysis are integrated, and the same analysis result as when one image data is sequentially processed from the beginning to the end is output.

As a concrete means, a video input means for inputting and storing video data and time division of the video data input by the video input means according to a predetermined condition to generate a plurality of partial videos. Video dividing means, a video analyzing means for executing parallel scene information analysis for each partial video obtained by the video dividing means, and a plurality of analysis results obtained by the video analyzing means are integrated. It has an analysis result integrating means and an analysis result outputting means for outputting the analysis result obtained by the analysis result integrating means.

By using the above means, the input video data is time-divided and a plurality of partial videos are generated, whereby the video analysis processing can be executed in parallel. By integrating multiple analysis results that are generated as a result of parallel execution, the video analysis time can be shortened compared to the conventional method in which the comparison calculation processing of the frame images that make up the video is performed sequentially from the beginning to the end of the video. Becomes

However, in the process of time-dividing the video data, if the video is simply divided at a certain arbitrary time as a boundary, the comparison calculation process of the frame images before and after the division point is not performed, so that the vicinity of the division point The correct video analysis result of may not be obtained. For example, if the point at which the video is divided is at the same timing as the scene switching, the frame images before and after the scene switching are included in different partial videos, and the two cannot be compared. Unable to extract scene change information.

Such a decrease in the image analysis accuracy may cause a problem when a more accurate image analysis result is required. Therefore, in the second aspect of the present invention, the image division means further inputs A dividing condition determining means for determining a dividing condition such as the number of divided divided image data or a dividing size, and a frame that overlaps temporally adjacent partial images among a plurality of generated partial images An overlapping frame number determining means for determining the number and a section determining means for determining a start frame number and an end frame number of each partial video for a plurality of generated partial videos are provided, and the analysis result integrating means includes: Analysis result reading means for reading the analysis result of each partial image obtained by the image dividing means, and overlap between temporally adjacent partial images set by the image dividing means Comparing the analysis result of the frame portion that is provided with an integrated determination means for performing integration of the analysis results.

In this way, when the video is divided, the adjacent partial videos have overlapping frame images, and the analysis results of the overlapping parts are compared and integrated so that the video generated by dividing the video It is possible to prevent a decrease in analysis accuracy and complete the video analysis processing in a shorter time than before.

The above means can be realized by a computer and a software program installed and executed in the computer. The program is a computer-readable portable medium memory,
It can be stored in an appropriate recording medium such as a semiconductor memory or a hard disk.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION An example of a video analysis device to which the present invention is applied will be described below with reference to the drawings. Figure 1
Shows a configuration example of a video analysis device according to the present invention. The video analysis device 1 includes a video input unit 11, a video division unit 12, N video analysis units (1 to N) 13, an analysis result integration unit 14, and an analysis result output unit 15.

The video input unit 11 is an input device for storing video data input from the outside in a memory or the like, and can be realized by a video capture board or the like. The input video data includes analog video signals such as TV broadcasts and digital data recorded on a recording medium such as a CD-ROM and a DVD.

The video division unit 12 reads the attribute information (file size, frame rate, total number of frames, etc.) of the video data input by the video input unit 11, and the number of partial videos generated and the time of the partial video generated. Determine conditions such as length and file size of partial video, calculate the start frame number and end frame number of the partial video to be generated using the attribute information of the read video data and the determined conditions as parameters, and generate the partial video The CP is a processing device that passes the generated partial video to the video analysis unit (1 to N) 13.
It can be realized by U or the like.

Since the partial image can be generated by the image dividing unit 12, the image analyzing units (1 to N) 13 can execute the image analyzing process of each partial image in parallel. In addition,
For example, when the image data is in a digital file format on a computer, the attribute information of the image data exists as header information of the file, and can be obtained by reading this with a dedicated software program.

The video analysis units (1 to N) 13 execute the analysis processing of the scene information of each partial video generated by the video division unit 12 in parallel, and the analysis result of each partial video is analyzed by the analysis result integration unit 1.
4 is a processing device to be delivered to the CPU 4 and can be realized by a processing device such as a CPU. As a method of determining which partial video each video analysis unit (1 to N) 13 processes, a video file to be processed can be specified from the video division unit 12 to the video analysis unit (1 to N) 13. It is possible to apply the method of notifying such a message. Alternatively, a method of assigning sequential numbers to the video analysis units (1 to N) 13 and partial video files in advance and each video analysis unit (1 to N) 13 reads out a file having the same file number as itself. is there.

The video analysis method used in the video analysis unit (1 to N) 13 compares differences between frame images constituting video, as used in the document cited in the above-mentioned "Prior Art". It can be realized by using a method (reference document [1]), a method of comparing changes in audio signals accompanying frame image sequences (reference document [3]), or a method of using both of them together. In addition, the analysis result is obtained by counting the number of scenes having a specific characteristic, expressing the section by time information (start time, end time), or the like, or a frame image of the scene in which the characteristic is detected by analysis. Is stored and a list of the image files is displayed.

The analysis result integration unit 14 includes a video analysis unit (1 to
N) is a processing device that receives the analysis result of the scene information of each partial video obtained by 13 and integrates it as one analysis result, and can be realized by a processing device such as a CPU. As a method of integrating the analysis results, the analysis results are sorted in the order of the numbers using the numbers set for each partial video.
There is a way to combine. The integrated processing of the analysis results of the video analysis unit 13 makes it possible to provide the same analysis result as one analysis result for one input video data.

The analysis result output unit 15 includes the analysis result integration unit 1
4 is an output device that reads out the analysis result obtained in 4 from the storage device and outputs it to a display device such as a computer display.

FIG. 2 shows an example in which the video analysis device shown in FIG. 1 is constructed as a video analysis system using a plurality of computers. The video analysis system using a plurality of computers includes a video division server 21, a video analysis server 22 (A to C), and an analysis result integration server 23. Each server is a computer having an arithmetic unit, a memory and a hard disk, and is connected by a network 30 so that data can be exchanged.

The video division server 21 is provided with the video input section 11 and the video division section 12 of the video analysis apparatus 1 of FIG. 1, and the analysis result integration server 23 is similarly analyzed by the video analysis apparatus 1 of FIG. The result integration unit 14 and the analysis result output unit 15 are arranged. In addition, the video analysis server 22 (A ~
C) is the video analysis unit 13 in the video analysis device 1 of FIG.
Has at least one function of. In this embodiment,
Although an example of using three video analysis servers 22 (A to C) is shown, the number of video analysis servers 22 may actually be any number. By increasing or decreasing the number of video analysis servers 22, the number of video analysis processes executed in parallel can be controlled.

FIG. 3 shows an example of video analysis contents and video analysis results in the present embodiment. FIG. 3A is an example of the video analysis content targeted by the present invention, and FIG.
Is an example of a video analysis result. As shown in Fig. 3 (a), the video analysis contents are, as shown in Fig. 3 (a), a section from one cut point to the next cut point (event type: cut), and a section where a telop is displayed in the video (event type : Telop), and a section in which music is flowing in the video (event type: music), three events are detected at the same time.

Therefore, as shown in FIG. 3 (b), the analysis result shows that, as the first item, which event of the above is detected, and as the second item, the event is a video image. In the example of this analysis result, each detected event is composed of a start time indicating how many seconds it started, and a third item, an end time indicating how many seconds the event ended. The start time of the event is listed from the top in ascending order.

FIG. 4 shows an example of the structure of the video division unit in the video analysis device of the present invention. The video division unit 12 determines the video division condition by a method such as an attribute information acquisition unit 121 that acquires video attribute information from the header information of the input video data and a method of reading a preset video division condition. The division condition determining unit 122, the overlapping frame number determining unit 123 that determines the number of frames that overlap with each other in temporally adjacent partial images among the generated partial images, and the attribute information acquiring unit 121. Image attribute information such as the acquired number of frames and the division condition determination unit 12
2 and a section determination unit 124 that determines the start frame number and the end frame number of each partial video using the division conditions such as the number of divisions determined by 2 and the number of overlapping frames determined by the number of overlapping frames determination unit 123 as parameters. To be done.

FIG. 5 shows an example in which partial images temporally adjacent to each other have an overlapping region (frame). In FIG. 5, the input video data is divided into three partial videos,
Each partial video has two overlapping frames. By the video dividing unit 12 having the configuration shown in FIG. 4, as shown in the video dividing example of FIG. 5, the temporally adjacent partial videos are input so as to have overlapping regions (frames). Video data can be divided into multiple partial videos.

FIG. 6 shows an example of the structure of the analysis result integration unit in the video analysis apparatus of the present invention. The analysis result integration unit 14
An analysis result reading unit 141 that reads the analysis result of each partial video output from the video analysis unit (1 to N) 13 and a section of each event in the read analysis result spans a plurality of partial videos. If it is judged that the event is one that spans multiple partial videos, it is possible to combine the events currently recognized as two or more different events into one event. Integrate the analysis results, 1
It is configured by the integrated determination unit 142 that takes one analysis result.

Next, in order to clarify the meaning of providing the overlapping area in the partial video, without providing the overlapping area in the partial video,
A problem that may occur when each partial video is simply divided at an arbitrary time will be described.

There are roughly two problems, and the first problem is that one event section (scene, etc.) may be divided into two event sections and detected. . In other words, an event that should not have been detected is detected. In order to prevent the detection of this unnecessary event, it is conceivable to combine two separated event sections into one by integrated processing.
On the other hand, if the event section is actually separated at the division point due to scene switching or the like, this separate event cannot be detected.

The second problem is a problem that occurs when video analysis is required to detect an event that requires a certain amount of time for detection. If it is divided by, the analysis time of either or both of the partial images may not be detected because the time required for detection is not satisfied. Is.

In order to avoid the above two problems, an overlapping area of partial images is provided. Hereinafter, the utilization method will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the overlapping area integration processing according to the present invention.

First, FIG. 7A is a diagram for explaining an example of integration of analysis results when one event section is divided into two event sections and detected. The analysis result A is 30 seconds from 200 seconds of the input video data.
We analyzed the number of seconds between each cut, telop, and music event included in the partial video up to 0 seconds (hereinafter, this partial video is called partial video A for convenience). The analysis result B is a result example, and an event included in a partial video of the same input video data as the analysis result A from 290 seconds to 400 seconds (hereinafter, this partial video is referred to as partial video B for convenience) is analyzed. This is a result example.

As can be seen from the fact that the end time of the last cut event of the partial analysis result A is 300 (unit is second) and the start time of the first cut event of the analysis result B is 290, The image A and the partial image B have an overlapping area for 10 seconds. That is, the analysis result of the video for 10 seconds is included in both the analysis result A and the analysis result B.

Here, the event of the telop detected at the end of the analysis result A (280 to 300) and the event of the telop detected as the second event of the analysis result B (300 to 310). ), The overlap area is 29
0 to 300 are included in the telop of the analysis result A, but not included in the telop of the analysis result B. Therefore, it can be determined that the telop of the analysis result A and the analysis result B are independent events.

On the other hand, when attention is paid to the last cut event (280 to 300) of the analysis result A and the first cut event (290 to 320) of the analysis result B, it is an overlapping area.
Since 90 to 300 are included in both events, it can be determined that this event is originally detected as one event.

In this way, whether or not each event is connected before and after dividing the image is compared with respect to all the events, and it is determined that the event is originally detected as one event. If so, combine the events. By this processing, the integrated analysis result A and analysis result B become the analysis result after integration.

Next, FIG. 7B shows an example of integration of analysis results when the time required for event detection is not satisfied due to image division. Similar to (a),
Analysis result A is 3 seconds from 200 seconds of the input video data.
This is an example of the result of analyzing how many seconds each cut, telop, and music event included in the partial video up to 00 seconds occurred, and the analysis result B is the same as the analysis result A. It is an example of the result of having analyzed the event contained in the partial video from 290 seconds to 400 seconds of input video data. The overlapping area is also set to 10 seconds as in the case of (a).

Here, attention is paid to the musical event (295 to 330) which is the second detected event of the analysis result B. In the analysis result B, the music event is detected in a part (295-300) of the overlapping area 290-300, but in the analysis result A, the music event is detected between 290-300. Not detected. In such a case, it is determined that the event originally detected in the detection result A does not satisfy the time required for detecting the event due to the image division, and the analysis content of the analysis result B is prioritized. Then, if the analysis results are integrated, the correct analysis result can be obtained.
Of course, the overlapping area must be provided for at least the time required for event detection.

As described above, by providing an appropriate overlapping area in temporally adjacent partial images, it is possible to suppress a decrease in image analysis accuracy that may occur due to image division.

FIG. 8 is a flowchart of integration processing by the analysis result integration unit. Further, FIG. 9 shows an example of the analysis result of the partial video, and shows an example of the analysis result report, the header table and the analysis result table. As shown in FIG. 9, the analysis result of each partial video is composed of a header table 40 indicating which part of the input video data is the report analyzed and an analysis result table 50 describing the actual analysis result. It is configured.

At the start, the counter i for designating the division number is 1. First, the analysis result is read from the analysis result file with the division number 1 and expanded in the memory (step S10), and the division number N is read from the header table 40.
Is read (step S11), and N-1 empty comparison event tables (FIG. 10) are created (step S1).
2).

Next, the analysis result is read from the analysis result file with the division number 2 and expanded on the memory (step S1.
3), the event is extracted and recorded in the created comparison event table (step S14). In the same manner, step S1 is sequentially performed by sequentially adding 1 to i until the division number N.
The process of S3 and S14 is repeated (step S15).

Here, FIG. 10 shows an example of creating a comparison event table. The comparison event table 60 extracts the start time and end time of the first detected event of each event in the partial video analysis result, and stores the extracted start time and end time for each event. In the example of FIG. 10, the first event {cut, 280-300} from the top in the analysis result is the first cut event, which is the comparison event table 60.
It is stored in. Next, the third event from the top {telop, 280-310} is the event of the first telop, and this is stored in the comparison event table 60. Next, the fifth event from the top {Music, 310-
420} is the first music event, which is stored in the comparison event table 60. The comparison event table 60 is created for each analysis result.

After the second to Nth comparison event tables 60 are created, the counter i is returned to 1 (step S16), and the analysis results from the analysis result file of division number 1 are sequentially read into the memory (step S16). S1
7). The comparison event table 60 is referred to for each event regarding the read analysis result, and the comparison event table 6
If each event in 0 and the start time and end time is independent, it is copied as it is to the integrated analysis result table, and if the event sections overlap, the events are combined,
Write to integrated analysis result table (step S1)
8). The processing of steps S17 and S18 is divided into division numbers N-1.
(Step S19).

Finally, the analysis result of the division number N is read (step S20), the analysis result of the portion not written in the integrated analysis result table is added as it is to the integrated analysis result table (step S21), and the final integrated analysis is performed. Write the result table as a file (step S2)
2).

Next, the video analysis device 1 converts the analog signal photographed by TV broadcasting or a video camera into A /
An example of performing video analysis while performing D conversion and storing in a storage device such as a hard disk will be described. This is because the processing speed of A / D converting the video into a file is several times faster than the analysis speed of the video, and the processing time is very long so that the processing cannot catch up with the input. Here is an example. The system configuration for this example is
It is similar to FIG.

The video input unit 11 A / D converts an analog signal such as an input TV broadcast and stores it in a storage device such as a hard disk. When the storage of the input signal starts, the video division unit 12 instructs the video analysis unit (1) 13 to start the analysis of the video of the already stored portion. When data of a predetermined time length is assigned to one video analysis unit (1) 13, the next video analysis unit (2) 13 is instructed to start analysis of the video data to be stored. Similarly, when data of a predetermined time length is assigned, the next video analysis unit (3) 13 (not shown) is instructed to start analysis of the video data to be stored. By repeating such processing, when the analysis of the initially assigned data is completed,
New data is again assigned to the video analysis unit (1) 13.

Each of the video analysis units (1 to N) 13 having received the stored video data analyzes the video and sends the analysis result to the analysis result integration unit 14. Analysis result integration unit 14
Receives the analysis results transmitted from the video analysis units (1 to N) 13 in sequence and performs integration processing. The integration process is continuously performed, but when it is determined that the analysis result is not affected by the analysis results of other partial images, the analysis result output unit 15 sequentially outputs the results to a display device such as a computer display or a printer. . In this example, by determining the number of video analysis units 13 so that the input of the video signal and the output of the analysis result are balanced, the video analysis can be performed in real time even in the video analysis with high processing cost.

[0051]

As described above, the video analysis processing can be executed in parallel by time-dividing the input video data and generating a plurality of partial videos.
By integrating multiple partial analysis results resulting from parallel execution of video analysis processing, the video analysis time is longer than that of the conventional method in which the comparison calculation processing of the frame images that make up the video is performed sequentially from the beginning to the end of the video. Can be shortened.

Furthermore, when dividing an image, the adjacent partial images are made to have an overlapping region, and the analysis results of the overlapping regions are compared and integrated to thereby cause an image analysis that may occur by dividing the image. It is possible to prevent a decrease in the accuracy of.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of a video analysis device according to the present invention.

FIG. 2 is a diagram showing an example in which a video analysis device is constructed as a video analysis system using a plurality of computers.

FIG. 3 is a diagram showing an example of video analysis contents and video analysis results.

FIG. 4 is a diagram illustrating a configuration example of a video analysis unit.

FIG. 5 is a diagram showing an example in which partial images temporally adjacent to each other have an overlapping region (frame).

FIG. 6 is a diagram showing a configuration example of an analysis result integration unit.

FIG. 7 is a diagram illustrating an overlapping area integration process.

FIG. 8 is a flowchart of integration processing by an analysis result integration unit.

FIG. 9 is a diagram showing an example of an analysis result report, a header table, and an analysis result table.

FIG. 10 is a diagram showing an example of creating a comparison event table.

[Explanation of symbols]

1 Video analysis device 11 Video input section 12 Video division 121 Attribute information acquisition unit 122 Division condition determination unit 123 Overlap frame number determination unit 124 section determination unit 13 Video analysis unit (1 to N) 14 Analysis result integration section 141 Analysis result reading section 142 Integrated judgment unit 15 Analysis result output section

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Masashi Morimoto             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation (72) Inventor Haruhiko Kojima             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation F-term (reference) 5C023 BA01 CA01 DA04 DA08                 5C052 AA17 AC08 GA01 GA03 GA04                       GA07 GB01                 5C053 FA14 FA21 FA23 HA08 HA29                       HA33 JA24 KA03 KA24 KA30                       LA01 LA06 LA11 LA14                 5L096 HA02

Claims (6)

[Claims] 1. A video analysis device for analyzing an input video signal and outputting a result of the analysis, the video input means for inputting and storing video data, and the video data input by the video input means. A time division according to a predetermined condition, and an image division means for generating a plurality of partial images,
Video analysis means for executing parallel analysis of scene information for each partial video obtained by the video division means, and analysis result integration means for integrating a plurality of analysis results obtained by the video analysis means, An image analysis device, comprising: an analysis result output means for outputting an analysis result obtained by the analysis result integration means. 2. The image analysis device according to claim 1,
The video splitting means determines splitting conditions such as a splitting number or a splitting size of the input video data, and splits the generated partial videos into partial videos that are temporally adjacent to each other. On the other hand, the analysis includes the overlapping frame number determining means for determining the number of overlapping frames and the section determining means for determining the start frame number and the end frame number of each partial video for a plurality of generated partial videos. The result integrating means reads the analysis result of each partial video obtained by the video dividing means, and the analysis result reading means, and analyzes the overlapping frame part of the temporally adjacent partial videos set by the video dividing means. An image analysis apparatus comprising: an integrated determination unit that compares results and integrates each analysis result. 3. A video analysis method for analyzing an input video signal and outputting a result of the analysis, the video input step of inputting and storing video data, and the video data input in the video input step. A time division according to a predetermined condition to generate a plurality of partial videos, and a video analysis step for executing in parallel the scene information analysis for each partial video obtained in the video division step. And an analysis result integration step of integrating a plurality of analysis results obtained by the video analysis step, and an analysis result output step of outputting the analysis result obtained by the analysis result integration step. analysis method. 4. The image analysis method according to claim 3,
The video division step includes a division condition determination step of determining a division condition such as the number or size of division of the input video data, and a step of dividing a plurality of generated partial videos into temporally adjacent partial videos. On the other hand, the method further includes a step of determining the number of overlapping frames for determining the number of overlapping frames, and a step of determining a start frame number and an end frame number of each partial video for a plurality of generated partial videos. The result integration step includes an analysis result reading step of reading an analysis result of each partial video obtained by the video division step, and an analysis of overlapping frame parts of temporally adjacent partial videos set in the video division step. An image analysis method comprising: an integrated determination step of comparing results and integrating each analysis result. 5. A video analysis program for causing a computer to execute the steps of the video analysis method according to claim 3 or 4. 6. A recording medium for a video analysis program, wherein a program for causing a computer to execute the steps in the video analysis method according to claim 3 or 4 is recorded.