JP2010176240A - Video analysis device, video analysis program, video analysis control device, and video analysis control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve a video analyzing device which is excellent in versatility and scalability by allowing a control part to transfer a frame required by each video analyzing part by uniform processing. <P>SOLUTION: A video analyzing part 10 has a featured value output part 12 for achieving each featured value extraction technology, and a next-frame designation part 11 for designating next-frame information as a frame number or frame time information required the next each time image data are successively input. A control part 14 transfers frame data to the video analyzing part 10 only when the frame designated by the next-frame designation part 11 of each video analyzing part 10 is obtained by a frame data acquisition part 13. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a video analysis device, a video analysis control device, and programs for analyzing various feature amounts from video data.

  There have been various techniques for analyzing video data and automatically outputting characteristic sections of the video and their associated values (hereinafter referred to as “features”).

  Patent Document 1 describes a technique for detecting a video cut point from a video data sequence. By this method, a cut point (cut section) including a scene change that is slow in time can be output.

  Patent Document 2 describes a technique for estimating camera parameters from an image data sequence (described in the same meaning as video data). By this method, camera operation called panning, tilting, etc. is performed from video data. Can be output along with its parameter values for a certain interval.

  Patent Document 3 describes a character region extraction technique for extracting a character portion from a frame image as a pixel connection region. By applying this method to a frame image of video data, a general telop is obtained from the video data. A video section including a character area called “Tel” and its telop position information can be output.

  Patent Document 4 describes a technique for detecting an animal up-frame image (also referred to as an up-shot) from a video. By this method, a video frame section in which a moving object is close-up and relatively large is shown. Can be output.

  What is common to these technologies is that video is treated as continuous image data (called a frame), and multiple frames are used as input data, and feature quantities such as cuts, camera work, telops, and animal up-shots are added. It is to calculate and output. Therefore, combining these technologies with a single frame data acquisition unit (function to acquire video as frame data), a video analysis device or video that simultaneously outputs various feature quantities to a single video data. It was possible to construct an analysis program. Actually, the process of acquiring the frame data from the video consumes a large amount of resources on the computer such as a CPU and a memory. Therefore, the advantage of extracting a plurality of feature amounts by one decoding process is great.

  Note that video data is composed of many frame images. For example, in standard 30 fps (frame / second) video data, each frame is an image of about 33 msec (millisecond). When the techniques described in Patent Documents 1, 2, 3, 4 and the like described above are realized, a large amount of resources necessary for calculation such as a CPU and a memory are required. Therefore, in order to increase the execution speed and execution efficiency, 33 msec. In many cases, the calculation is not performed for every frame, but frame images are acquired at intervals of 100 msec or 500 msec, and extraction processing is performed by each technique.

  For example, a very short cut interval of 1 second or less is not something that can be recognized by the user in the first place, but rather should be treated as instantaneous noise. For this reason, even if cut point extraction processing is performed on the basis of a frame every 300 msec, the practical problem is unlikely to occur.

  The same applies to camera work, and in particular, the feature extraction of camera work requires a relatively large amount of resources rather than cutting, so it is more practical to further increase the section and process every 500 msec.

  On the other hand, moving object up-shot detection searches for a frame that captures the subject most closely, so that the user can just frame the video during the period when it is determined that there is a moving object. However, as in the case of searching, all frames every 33 msec are processed as input targets.

  In the case of a telop, as described in paragraphs [0031] to [0037] of Patent Document 3, a plurality of temporally continuous frame images are input to create an averaged color image, thereby It is known that there is an effect of suppressing the false detection of. For this reason, for example, it is necessary to control such that four consecutive images are processed every 500 msec.

Japanese Patent No. 2839132 Japanese Patent No. 3408117 Japanese Patent No. 3467195 JP 2006-244074 A

  As described above, there are a variety of technologies that treat video as consecutive frames and output feature values using multiple frames that are close in time as input data. Extraction technology is expected to be developed.

  However, it becomes difficult to configure a device or a program for simultaneously outputting various feature amounts by combining the video analysis device or the video analysis program that realizes these individual feature amount extraction techniques. In other words, since each feature extraction technique requires frame data at different intervals, a device or program that controls the whole must call each video analysis unit while managing each interval. Don't be. Furthermore, when an unknown feature quantity extraction technique is developed, the control unit that is a device or program for controlling the whole must be modified every time accordingly.

  FIG. 7 is a diagram for explaining the problem of the present invention, and is a diagram illustrating a configuration example of an apparatus that simultaneously outputs various feature amounts by combining individual feature amount extraction techniques. FIG. 8 is a process flowchart of the control unit shown in FIG.

  When considering a device that outputs various feature amounts simultaneously by combining a video analysis device that realizes individual feature amount extraction of the prior art, it can be considered that the device configuration is as shown in FIG. 7, for example. In FIG. 7, a CUT video analysis unit 101 detects a cut point as described in Patent Document 1, and a feature amount related to the cut point (hereinafter referred to as “CUT feature amount”) as a result of the detection. The CUT feature quantity output unit 102 having the output function is provided.

  The CAM video analysis unit 111 estimates a camera parameter as described in Patent Document 2, and has an output function of a feature value related to camera work (hereinafter referred to as “CAM feature value”) as a result of the estimation. A CAM feature amount output unit 112 is provided.

  The TLP video analysis unit 121 detects a telop as described in Patent Document 3, and has a function of outputting a feature amount related to a telop (hereinafter referred to as “TLP feature amount”) as a detection result. A feature amount output unit 122 is included.

  The UPS video analysis unit 131 detects a moving object up-frame image as described in Patent Document 4, and a feature amount related to the moving object up-frame image (hereinafter referred to as “UPS feature amount”). A UPS feature amount output unit 132 having an output function.

  The frame data acquisition unit 140 decodes the compressed and encoded video data into a frame image, and passes the decoded frame data to the control unit 150. The control unit 150 passes the frame data obtained from the frame data acquisition unit 140 to the CUT video analysis unit 101, the CAM video analysis unit 111, the TLP video analysis unit 121, and the UPS video analysis unit 131 at appropriate timings. Perform processing to analyze the video.

  Processing performed by the control unit 150 will be described with reference to FIG. In FIG. 8, TScut is a memory area for holding a time stamp [msec] to be processed next by the CUT video analysis unit 101, and TScam is a time stamp to be processed next by the CAM video analysis unit 111. A memory area TStlp for holding is a memory area for holding a time stamp to be processed next by the TLP video analysis unit 121. Since the UPS video analysis unit 131 analyzes all frames, it is not necessary to use a memory area (TSups) for holding a time stamp to be processed next. Ctlp is a memory area for holding a counter of the number of frame images to be processed successively thereafter.

  The control unit 150 first initializes each function such as opening a video file storing video data to be analyzed and securing a buffer area (step S101). Next, all the values of the time stamp variables TSxxx and the counter Ctlp are set to 0 (step S102). Thereafter, frame data is acquired from the frame data acquisition unit 140 (step S103).

  Subsequently, the control unit 150 repeats the processing from step S105 to step S116 described below and the processing of step S103 until the data of the video file is completed (step S104).

  Let the time stamp of the acquired frame be t [msec]. First, it is determined whether or not the time stamp t is TScut (step S105). If not TScut, the process proceeds to step S108. If TScut is reached, the frame data is transferred to the CUT video analysis unit 101 (step S106). Thereafter, a value obtained by adding 300 to the time stamp t is set in TScut (step S107). That is, the next processing target frame is set to a frame after 300 msec.

  Subsequently, it is determined whether or not the time stamp t becomes TScam (step S108). If not TScam, the process proceeds to step S111. When TScam is reached, the frame data is transferred to the CAM video analysis unit 111 (step S109). Thereafter, a value obtained by adding 500 to the current time stamp t is set in TScam (step S110).

  Next, it is determined whether Ctlp is greater than 0 or whether the time stamp t is TStlp (step S111). If the determination result is “false”, the process proceeds to step S116. If the determination result is “true”, the frame data is passed to the TLP video analysis unit 121 (step S112). Thereafter, it is determined whether or not Ctlp is smaller than 3 (step S113). If Ctlp is smaller than 3, Ctlp is incremented by 1 (step S114). Otherwise, a value obtained by adding 500 to the current time stamp t is set in TStlp, and 0 is set in Ctlp (step S115).

  The frame data of the acquired frame is passed to the UPS video analysis unit 131 (step S116). The UPS video analysis unit 131 analyzes all frames. Thereafter, the control unit 150 returns to the process of step S103, acquires the frame data of the next frame from the frame data acquisition unit 140, and repeats the process in the same manner.

  If the video file is finished (step S104), the end processing of each function such as releasing the buffer area and closing the video file is performed (step S117).

  As is clear from the above processing contents, each video analysis unit 101 is included in the flow of the control unit 150 simply by adding the control unit 150 for passing frame data to the conventional various video analysis units 101 to 131. Processing unique to ˜131 is entered, and complicated control is required. For this reason, the control unit 150 cannot perform uniform and uniform processing. This greatly impairs the expandability of such a video analysis apparatus or video analysis program as a whole.

  The present invention solves the above-mentioned problems and provides a mechanism that allows each video analysis unit to pass a necessary frame to the various video analysis units in a uniform process. The purpose is to make it possible to construct a video analysis system with excellent performance.

  In order to solve the above problems, in the present invention, the video analysis unit is provided with a next frame designation function as well as a feature quantity output function realizing each feature quantity extraction technique, and the control unit is designated by the next frame designation function. Only when the received frame is acquired from the frame data acquisition unit, the frame data is passed to the video analysis unit.

  Specifically, the video analysis apparatus according to the present invention includes a frame data acquisition unit that acquires video as image data for each frame, and sequentially inputs the image data for each frame, so that a characteristic section of the video or an accompanying image is obtained. A feature amount output unit that outputs a feature amount that is a value to be processed, and a next frame designation unit that designates next frame information that is frame number information or frame time information that is required next every time image data is sequentially input Or a plurality of video analysis units, and a control unit that obtains image data of a necessary frame from the frame data acquisition unit based on the next frame information specified by the video analysis unit and sends the image data to the video analysis unit. Is provided.

  Each of these units can be realized by a single computer, but it is also possible to adopt a device configuration in which the video analysis unit and the control unit are mounted on different computers connected by a network. At this time, the frame data acquisition unit can be mounted on the same computer as the control unit, or can be mounted on a different computer.

  According to the present invention, the control unit can realize a mechanism for passing a necessary frame to each video analysis unit without knowing in advance the acquisition interval of frame data unique to each video analysis unit. Therefore, even when an unknown feature quantity extraction technique is developed, it is possible to easily realize a mechanism for outputting a plurality of feature quantities at the same time without special modification of the control unit.

It is a figure which shows the structural example of the video analysis apparatus which concerns on embodiment of this invention. It is a process flowchart of a frame data acquisition part. It is a process flowchart of a control part. It is a process flowchart of the image | video analysis part which detects a cut point. It is a process flowchart of the image | video analysis part which detects a telop. It is a figure which shows the example of the video-analysis apparatus comprised with a network system. It is a figure which shows the structural example of the apparatus explaining the subject of this invention. It is a process flowchart of the control part shown in FIG.

  Below, the case where each part of this invention is mainly implemented using computer hardware and software programs, such as CPU and memory, is described as an example.

  FIG. 1 shows a configuration example of a video analysis apparatus according to an embodiment of the present invention. The video analysis apparatus 1 includes a plurality of video analysis units 10 [0], 10 [1], 10 [3],... That extract different feature amounts from the video data, and the analysis target video as image data for each frame. The frame data acquisition unit 13 to be acquired and the control unit 14 are configured. Since the video analysis apparatus 1 has versatility, even if the video analysis unit 10 is not plural but single, it functions normally.

  Each video analysis unit 10 includes next frame designation units 11 [0], 11 [1], 11 [2],..., Feature amount output units 12 [0], 12 [1], 12 [2],. … And.

  The next frame designation unit 11 has a function of notifying the control unit 14 of next frame information of a frame number or frame time information that is necessary next time a frame is sequentially input to the video analysis unit 10. The feature amount output unit 12 has a function of outputting feature amounts such as a characteristic section of the video and a value associated therewith by sequentially inputting image data for each frame.

  Based on the next frame information specified by the video analysis unit 10, the control unit 14 has a control function that each video analysis unit 10 acquires necessary frame data from the frame data acquisition unit 13 and sends it to the video analysis unit 10. Have.

  In this example, the video analysis unit 10, the frame data acquisition unit 13, and the control unit 14 according to the present embodiment are configured by software modules executed on a single computer, and operations between components are performed. Is described as being executed in the form of a function call for that software.

  The frame data acquisition unit 13 is configured by a software module having a function of inputting a file that has been subjected to processing such as encoding and compression in a format such as MPEG and decoding it into a frame image. Many such softwares have been implemented.

  FIG. 2 shows a processing flowchart of the frame data acquisition unit 13. As shown in FIG. 2, the frame data acquisition unit 13 mainly includes phases of “initialization”, “frame acquisition”, and “finalization”, and is called by sequential function calls from higher-level software (control unit 14). The function is provided by Specifically, it is as follows.

  First, when the control unit 14 specifies the target video file name as an argument and calls the “initialization” function, the frame data acquisition unit 13 opens the file specified by the argument (step S10), and the frame An initialization operation such as securing a necessary buffer area in the data acquisition unit 13 (step S11) is performed.

  Thereafter, the “frame acquisition” function call using the storage area address in the control unit 14 as an argument is continuously performed from the control unit 14, and the frame data acquired from the video file is decoded each time the frame data acquisition unit 13 is called. Are read one by one into the internal buffer area (step S12). When the video file is ended, the end of the file is notified to the function calling source control unit 14 (step S13). When frame data is obtained from the video file, the data for one frame is expanded to the designated storage area address in the control unit 14 (step S14) and returned to the control unit 14. The returned frame data includes temporal index information of a frame image in the video file called a time stamp. Depending on the implementation method of the frame data acquisition unit 13, not only temporal information but also index information having a number indicating the number of frames from the beginning may be included. In this case, the frame rate (per unit time) is included. By multiplying by the reciprocal of the number of frames, it can be handled in the same way as a time stamp.

  When the video analysis is complete and the control unit 14 calls the “terminate” function, the frame data acquisition unit 13 closes the video file (step S15) and releases the buffer area secured at initialization. (Step S16), and the process ends.

  A processing flow of the control unit 14 is shown in FIG. In the present invention, since it is possible to generalize a plurality of video analysis units 10 that output feature quantities using frames as input data, these N video analysis units 10 are referred to as a video analysis unit 10 [0] and a video analysis unit. 10 [1],..., Video analysis unit 10 [N-1]. Specifically, in accordance with the purpose of use, for example, the video analysis unit 10 [0] is a cut point detection video analysis unit, the video analysis unit 10 [1] is a telop detection video analysis unit,... And so on. Such a definition can be described, for example, in a setting file (not shown) of the video analysis apparatus 1 and read and specified when the control unit 14 is initialized.

  The control unit 14 has N arrays TS [] corresponding to the N video analysis units 10 in the memory area 15. Using this array value, in step S25 of FIG. 3, it is possible to determine whether or not to actually pass the frame data to each video analysis unit 10 and make a function call for analysis processing. If an analysis processing function call is made, the value of the array TS [i] is updated with the return value (or the output parameter value in another embodiment) in step S27. With this process, the frame required by the video analysis unit 10 can be managed by the control unit 14 for general use.

  The processing executed by the control unit 14 is as follows. First, each function is initialized (step S20). Here, the above-mentioned “initialization” function is called, an internal buffer area used as a frame data storage area is secured, and an initialization instruction is given to each video analysis unit 10 as necessary. Further, all the time stamp arrays TS [0] to TS [N−1] in the memory area 15 are set to 0 (step S21).

  Thereafter, the control unit 14 calls the “frame acquisition” function with the storage area address of the internal buffer area as an argument, and acquires frame data from the frame data acquisition unit 13 (step S22). It is determined whether the return value of the function call is “end of file” (step S23). If it is “end of file”, the process proceeds to step S29.

  If the return value is not “end of file” but “normally acquired” and the frame data is stored in the internal buffer area, the loop counter i is incremented from 0 to N−1 (step S24), and the following The processes in steps S25 to S27 are repeated until i = N−1 (step S28).

  Let the time stamp of the acquired frame be t [unit: msec]. In the iterative process, first, it is determined whether or not the time stamp t is equal to or greater than TS [i] (step S25). If t is smaller than TS [i], steps S26 and S27 are skipped. If the time stamp t is equal to or greater than TS [i], the frame data stored in the internal buffer area is transferred to the i-th video analysis unit 10 [i] (step S26). Thereafter, the value of the next frame information specified by the next frame specifying unit 11 [i] of the video analysis unit 10 [i] is set in the time stamp array TS [i] (step S27).

  If the above repeating process is completed up to i = N−1 (step S28), the process returns to step S22, and the acquisition of the next frame data is repeated.

  When “end of file” is notified from the frame data acquisition unit 13, the above-mentioned “end” function is called, the internal buffer area used as the frame data storage area is released, and each video analysis unit 10 is also necessary. End processing of each function such as an end processing instruction is performed (step S29), and the processing ends.

  Next, the processing flow of the video analysis unit 10 will be described according to the examples of FIGS. Here, an example using the video analysis technique of Patent Document 1 is shown in FIG. 4, and an example using the video analysis technique of Patent Document 3 is shown in FIG.

  The video analysis unit 10 [0] shown in FIG. 4 performs cut point detection by video analysis, and if there is an analysis processing instruction using one frame of frame data as an argument from the control unit 14, Patent Document 1 The cut point extraction process as described in (1) is executed (step S30). It is determined whether or not a cut point is detected by the cut point extraction process (step S31). If a cut point is detected, the feature amount output unit 12 [0] is called. Otherwise, the process proceeds to the next frame designation unit 11 [0] (step S33).

  The feature amount output unit 12 [0] outputs information of the detected cut point to a predetermined result file (or result storage memory area) (step S32).

  The next frame specifying unit 11 [0] sets a value obtained by adding 300 [msec] to t as next frame information (step S33), and returns the next frame information to the control unit 14 as a return value. By notifying the control unit 14 of the next frame information as t + 300, it becomes possible to perform a cut point detection process on a frame every 300 msec.

  The video analysis unit 10 [1] shown in FIG. 5 detects telops by video analysis. The initial value of the counter Ctlp prepared in the memory area 16 is set to zero. In this example, four continuous frame images are input every 500 msec, an averaged color image is created, and a telop is detected.

  When there is an analysis processing instruction using frame data for one frame as an argument from the control unit 14, it is first determined whether or not the value of the counter Ctlp is smaller than 3 (step S40). If it is smaller than 3, multiple frame averaging processing is executed by the method described in Patent Document 3 (step S41). On the other hand, when the value of the counter Ctlp becomes 3, the telop detection process by the method described in Patent Document 3 is executed (step S42). When a telop is detected (step S43), the feature amount output unit 12 [1] is called, and information on the detected cut point is output to a predetermined result file (or result storage memory area) (step S44). .

  Thereafter, the process proceeds to the process of the next frame specifying unit 11 [1], and it is determined whether or not the value of the counter Ctlp is smaller than 3 (step S45). If it is smaller than 3, the value of the counter Ctlp is incremented by 1 (step S46), a value obtained by adding 1 [msec] to t is set as the next frame information (step S47), and the next frame information is returned as the return value. To the control unit 14. In this way, by setting the next frame information to t + 1, if the time stamp advances even a little from the present time, the video analysis unit 10 [1] is called again from the control unit 14. Data will be passed.

  On the other hand, if the value of the counter Ctlp becomes 3, the value of the counter Ctlp is returned to 0 (step S48), and a value obtained by adding 500 [msec] to t is set as the next frame information (step S49). The next frame information is returned to the control unit 14 as a return value.

  As described above, the next frame information is notified to the control unit 14, so that four frames of data can be input every 500 msec, and a telop detection process can be performed.

  Here, an example has been described in which the next frame information specified by the next frame specifying unit 11 of the video analysis unit 10 is returned to the control unit 14 as a return value of an analysis processing function that calls the video analysis unit 10. As an embodiment of the above, it is also possible to adopt a method in which an output parameter is provided in the analysis processing function call, the next frame information is set in the output parameter, and the process returns.

  The example of the video analysis unit 10 for cut point detection and telop detection has been described. Similarly, for the video analysis unit 10 that extracts other types of feature amounts, the next frame designation unit 11 similarly uses the next frame number required. By providing means for notifying the control unit 14 of next frame information such as frame time information and the like, the control unit 14 can uniformly execute the delivery processing of the frame data to the video analysis unit 10.

  In the above embodiment, in order to make the explanation easy to understand, the example in which the functions of the respective units shown in FIG. 1 are implemented in a single computer has been described. However, in another embodiment, each function is provided in a separate computer. Implemented by the software modules executed above, these computers can also communicate with each other over a LAN.

  FIG. 6 shows an example in which a video analysis apparatus is constructed using a network system. The video analysis device according to the present embodiment includes a video analysis control device 20 realized by different computers (CPU and memory), video analysis processing devices 30 [0], 30 [1], 30 [2],. , And a network 40 such as a LAN. The video analysis control device 20 includes the frame data acquisition unit 13 and the control unit 14 described with reference to FIG.

  That is, the control unit 14 of the video analysis control device 20 is based on means for acquiring video from the frame data acquisition unit 13 as frame-by-frame image data in time order of frames, and next frame information specified by the video analysis processing device 30. , Means for determining whether or not the frame acquired from the frame data acquisition unit 13 is image data of a frame required by the video analysis processing device 30, and the frame acquired from the frame data acquisition unit 13 is the video analysis processing device 30. Is determined to be image data of a necessary frame, means for sending the image data of the frame to the video analysis processing device 30 and means for storing next frame information received from the video analysis processing device 30 are provided. .

  Each video analysis processing device 30 includes the video analysis unit 10 described with reference to FIG. The video analysis control device 20 and each video analysis processing device 30 transmit and receive commands and data such as frame data and next frame information through the network 40, as in the case where the video analysis device is realized by a single computer. The present invention can be implemented. Of course, it is possible to implement the present invention by mounting the frame data acquisition unit 13 and the control unit 14 on different computers.

DESCRIPTION OF SYMBOLS 1 Image | video analysis apparatus 10 Image | video analysis part 11 Next frame designation | designated part 12 Feature-value output part 13 Frame data acquisition part 14 Control part

Claims (4)

A frame data acquisition unit for acquiring video as image data for each frame;
By sequentially inputting image data for each frame, a feature amount output unit for outputting a feature amount which is a characteristic section of the video or a value associated therewith, and the image data are sequentially input each time the image data is sequentially input. One or a plurality of video analysis units having a next frame designating unit for designating next frame information which is a frame number or frame time information,
Based on the next frame information specified by the video analysis unit, each of the video analysis units acquires image data of a necessary frame from the frame data acquisition unit and sends the frame data to the video analysis unit;
A video analysis apparatus comprising: Computer
A frame data acquisition unit for acquiring video as image data for each frame;
By sequentially inputting image data for each frame, a feature amount output unit for outputting a feature amount which is a characteristic section of the video or a value associated therewith, and the image data are sequentially input each time the image data is sequentially input. One or a plurality of video analysis units having a next frame designating unit for designating next frame information which is a frame number or frame time information,
Based on the next frame information specified by the video analysis unit, each of the video analysis units acquires the image data of the necessary frame from the frame data acquisition unit and sends it to the video analysis unit,
Video analysis program to make it function. By sequentially inputting image data for each frame, a feature amount output unit for outputting a feature amount which is a characteristic section of the video or a value associated therewith, and the image data are sequentially input each time the image data is sequentially input. Sending image data of a frame to be analyzed to one or a plurality of video analysis processing devices including a video analysis unit having a next frame information unit for specifying next frame information which is frame number or frame time information, A video analysis control device for analyzing video,
Means for acquiring video as image data for each frame in the time order of the frame from the frame data acquisition unit provided in the video analysis control device or the external device;
Means for determining whether the frame acquired from the frame data acquisition unit is image data of a frame required by the video analysis processing device based on next frame information designated from the video analysis processing device;
Means for sending the image data of the frame to the video analysis processing device when it is determined that the frame acquired from the frame data acquisition unit is image data of a frame required by the video analysis processing device;
Means for storing next frame information received from the video analysis processing device;
A video analysis control apparatus comprising:   A video analysis control program for causing a computer to function as each means included in the video control device.

Images