JP4214990B2 - Event detection method, apparatus and program - Google Patents

Description

  The present invention relates to a linear pattern identification method, apparatus, and program, and more particularly, to an event detection method, apparatus, and program for detecting a specific event from an image.

Research has been conducted for a long time to improve the listability by detecting events in video. For example, there is a method of summarizing by detecting a pitching event from a baseball video (for example, see Patent Document 1).
JP 2003-52003 A

  However, in the above-described conventional method, the pitch event is specified by detecting the green region or the brown region in the video and detecting the composition when the pitcher throws. That is, detection by template matching.

  However, there are grounds with no turf in the infield, and the sand color of the mound varies from reddish brown grounds to common sand colors. Furthermore, it is a day game or a night game, and the way it is projected changes variously. Furthermore, the position where the camera is installed varies depending on the stadium and broadcasting station, and it is very difficult to prepare a template that can be used at all stadiums, game times, and broadcasting stations.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an event detection method, apparatus, and program capable of detecting an event with high accuracy without being influenced by a shooting environment from a video. To do.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is an event detection method for detecting a specific event from a video.
A video reading step (step 1) for reading video from the video storage means;
An acoustic feature detection step (step 2) for detecting acoustic features in the read video;
An event candidate detection step (step 3) for detecting an event candidate based on the video section in which the acoustic feature is detected;
A video feature extraction step (step 4) for extracting video features in the video segment of the event candidate;
A clustering step (step 5) for performing unsupervised clustering of video candidates based on video features;
A correct cluster selection step (step 6) of storing the upper cluster having a large number of elements as a correct cluster in the storage means is performed.

In the present invention (Claim 2), after the correct cluster selection step (Step 6),
A template image acquisition step for acquiring a template image from the result of clustering;
An event re-retrieval step for re-searching the entire video of the video storage means by using the template image is further performed.

Further, the present invention (Claim 3) is the acoustic feature detection step (Step 2).
A portion having a sudden sound feature in the sound power transition is extracted as a sound feature.

Further, the present invention (Claim 4) is, in the video feature extraction step (Step 4),
The RGB value of the image in the event candidate section is used as the video feature.

Further, the present invention (Claim 5) provides a video feature extraction step (Step 4).
The direction component of the edge image is used as the video feature.

Further, the present invention (Claim 6) provides a video feature extraction step (Step 4).
Motion features are used as video features.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 7) is an event detection device for detecting a specific event from a video,
Video storage means 110 for storing video;
Video reading means 1 for reading video from the video storage means 110;
Acoustic feature detection means 2 for detecting acoustic features in the read video;
Event candidate detection means 3 for detecting event candidates based on the video section in which the acoustic features are detected;
Video feature extraction means 4 for extracting video features in a video section of event candidates;
Clustering means 5 for performing unsupervised clustering of video candidates based on video features;
Correct cluster selection means 6 for storing the upper cluster having a large number of elements in the storage means 120 as a correct answer cluster.

Further, the present invention (Claim 8) includes a template image acquisition means for acquiring a template image from the result of clustering;
Event re-searching means for re-searching the entire video in the video storage unit 110 using the template image is further included.

Further, the present invention (Claim 9) is the acoustic feature detection means 2,
A portion having a sudden sound feature in the sound power transition is extracted as a sound feature.

Further, the present invention (Claim 10) is the video feature extracting means 4,
The RGB value of the image in the event candidate section is used as the video feature.

Further, the present invention (claim 11) is the video feature extraction means 4,
The direction component of the edge image is used as the video feature.

Further, the present invention (Claim 12) is the video feature extraction means 4,
Motion features are used as video features.

The present invention (Claim 13) is an event detection program for detecting a specific event from a video,
A program for causing a computer to execute a method for realizing the event detection method according to claim 1.

  As described above, according to the present invention, in the process of matching a sports video (for example, baseball) and a video scene of hitting or catching, the scene extraction processing is first performed from the sound, and then the extracted video By performing the scene clustering process, it is possible to match the video scene of hitting or catching with higher accuracy than in the conventional technique. For this reason, it becomes possible to detect an event with high accuracy without being affected by the shooting environment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the present invention, in the event detection apparatus, first, as empirical knowledge, detection is performed using acoustic information that has less change due to shooting angle or the like compared to video, and then processing for narrowing down events using video features is performed.

  FIG. 3 shows the configuration of the event detection apparatus according to an embodiment of the present invention.

  The event detection apparatus shown in the figure includes a reading unit 1, an acoustic feature extraction unit 2, an event candidate detection unit 3, a video feature extraction unit 4, a clustering unit 5, a correct cluster selection unit 6, a template creation unit 7, and a template usage re-search. Unit 8, a video storage unit 110, and an event storage unit 120.

  The reading unit 1 reads a video from the video storage unit 110 and outputs it to the acoustic feature extraction unit 2.

  The acoustic feature extraction unit 2 extracts the acoustic feature of the read video and outputs it to the event candidate detection unit 3.

  The event candidate detection unit 3 detects an event candidate from the acoustic feature and outputs it to the video feature extraction unit 4.

  The video feature extraction unit 4 extracts video features from the sections that are event candidates and outputs them to the correct candidate clustering unit 5.

  The correct candidate clustering unit 5 performs clustering of event candidates based on the video feature amount, and outputs the result candidates to the correct cluster selecting unit 6.

  The correct cluster selection unit 6 selects a correct cluster from the clustering result and outputs it to the template creation unit 7.

  The template creation unit 7 acquires a template image from the clustering result.

  The template usage re-search unit 8 re-searches the entire video using the template image, and stores the result in the event storage unit 120.

  The video storage unit 110 is a storage medium such as a disk device, and stores video.

  The event storage unit 120 is a storage medium such as a disk device, and records a cluster event output from the template use re-search unit 8.

  FIG. 4 is a flowchart of the operation of the event detection apparatus in one embodiment of the present invention.

  Step 101) The reading unit 1 reads a video from the video storage unit 110.

  Step 102) The acoustic feature extraction unit 2 calculates the power of the acoustic signal of the read video, and extracts the acoustic feature based on the transition of the power. As a method for extracting acoustic features, there is a method for detecting a portion having a sudden sound feature in the transition of power.

  Step 103) The event candidate detection unit 3 detects an event candidate based on the acoustic feature extracted by the acoustic feature extraction unit 2. Specifically, a video section including the extracted acoustic features is detected.

  Step 104) The video feature extraction unit 4 obtains the video feature amount from the video section detected by the event candidate detection unit 3 by an arbitrary method.

  Step 105) The correct candidate clustering unit 5 performs video clustering based on the feature amount obtained by the video feature extraction unit 4. As a clustering method, there is a method of performing unsupervised clustering using video feature amounts.

  Step 106) The correct cluster selector 6 selects a correct cluster based on the clustering result.

  Step 107) The template creation unit 7 acquires a template image from the clustering result.

  Step 108) The template use re-search unit 8 re-searches the video storage unit 110 using the template image, and stores the result in the event storage unit 120.

  Note that the processing of step 107 and step 108 is not essential, and if a feature that appears to the extent that it can be used for redetection as a video feature amount (for example, MHI: Motion History Image) can be detected, a template is determined in step 107. Good results can be obtained by generating and re-searching in step 108 using the template.

  Embodiments of the present invention will be described below with reference to the drawings.

  In this embodiment, a candidate for a pitching sound or a catching sound generated after a pitch is obtained from a baseball video by acoustic analysis, a pitching event candidate is obtained, and clustering based on motion features from a moving image section for each pitching event candidate is obtained. An example in which a pitching event is acquired from the video will be described.

  FIG. 5 shows the configuration of an event detection apparatus according to an embodiment of the present invention.

  The event detection apparatus shown in the figure includes a video reading unit 10, a sudden sound segment detection unit 20, a sudden sound segment acquisition unit 40, a sudden sound segment video clustering unit 50, a template creation unit 70, a template image utilization reconstruction measure unit 80, a video. The storage unit 110 and the hitting ball event storage unit 120 are configured.

  The video reading unit 10 reads a video from the video storage unit 110 and outputs it to the sudden sound section detection unit 20.

  The sudden sound section detection unit 20 detects a portion including sudden sound from the read video and outputs the detected part to the sudden sound section video acquisition unit 40. At this time, sudden sound is detected by acoustic analysis to be described later, and pitching scene candidates are acquired.

  The sudden sound section video acquisition unit 40 obtains the sudden sound section video of the detected sudden sound section and outputs it to the sudden sound section video clustering unit 50.

  The sudden sound section video clustering unit 50 selects an event by clustering the sudden sound section video.

  The template creation unit 70 acquires a template image from the clustering result.

  The template use re-search unit 80 performs re-search using the template image, and stores the result in the hit ball event storage unit 120.

  FIG. 6 is a flowchart of event detection processing according to an embodiment of the present invention.

  Step 201) The video reading unit 10 reads a video from the video storage unit 110.

  Step 202) The sudden sound section detection unit 20 detects a part including sudden sound from the read video by acoustic analysis.

  Step 203) In the sudden section video acquisition unit 40, a video corresponding to the sudden sound section detected in Step 202 is acquired.

  Step 204) In the sudden sound section video clustering unit 50, the sudden sound section video acquired in step 203 is clustered, and an event is selected based on the clustering result.

  Step 205) The template creation unit 70 acquires a template image from the clustering result obtained in step 204.

  Step 206) The template use re-search unit 80 re-searches the video storage unit 110 using the template image, and stores the result in the hitting ball event storage unit 120.

  Hereinafter, the above operation will be described in detail.

  First, the sudden sound detection operation in step 202 will be described.

  FIG. 7 is a flowchart for detecting sudden sound according to an embodiment of the present invention.

  Step 301) The sudden sound section detection unit 20 calculates the power of the audio signal of the read video.

  Step 302) A portion having a sudden sound characteristic in the power transition is detected.

  Next, step 301 in FIG. 7 will be described in detail.

  FIG. 8 is a flowchart of a method for calculating the power of an acoustic signal according to an embodiment of the present invention.

  Step 401) The sudden sound section detection unit 20 initializes i = 0.

  Step 402) A window function is applied to the acoustic signal data [j] from j = i to j = i + WindowSize (window size). As the window function, Hamming window, Hanning window, rectangular window, etc. are famous. Here, any window function may be used.

  Step 403) With \ sum_ {j = i ^ {i + WindowSize} data [i], the power is calculated for the acoustic signal having a window size width from i and substituted for p (i).

  Step 404) Add WindowSlide (window slide amount) to i.

  Step 405) It is determined whether i + WindowSlide is smaller than DataMax. If it is smaller, the process proceeds to step 402; otherwise, the process is terminated.

  Next, the suddenness detection process in step 302 of FIG. 7 will be described in detail.

  FIG. 9 is a flowchart of sudden detection processing according to an embodiment of the present invention.

  Step 501) i = 0 is initialized.

  Step 502) Initialize flag = 0.

  Step 503) After assigning new_flag to the current flag, new_flag = F (p (i), flag) is calculated. F (p (i), flag) is a function determined from the power obtained in step 301 and the current flag. FIG. 10 shows how to obtain new_flag.

  Step 504) If new_flag is 1, the process proceeds to step 505, and if not, the process proceeds to step 506.

  Step 505) Substitute i for time st.

  Step 506) When flag = 3, if new_flag is 1 or 0, the process proceeds to Step 507; otherwise, the process proceeds to Step 510.

  Step 507) Substitute i-st for t.

  Step 508) If th4 <t <th5 is satisfied, the process proceeds to step 509; otherwise, the process proceeds to step 510.

  Step 509) The time st is stored as the head of the sudden sound section.

  Step 510) i + WindowSlide is added and the process proceeds to Step 503. Otherwise, the step process is terminated.

  Step 511) Add WindowSlide to i and go to Step 503.

  FIG. 10 is a table showing a function for obtaining new_flag. It is a shaded portion that is detected as a sudden sound. When th1> p (i) and flag = 3, new_flag = 1, and when th3> p (i)> th1 and flag = 3, new_flag = 0.

  Next, the thumbnail image clustering process in the sudden sound section video clustering unit 50 in step 204 will be described.

  FIG. 11 is a flowchart of thumbnail image clustering according to an embodiment of the present invention.

  Step 701) Features are extracted from the sudden sound section video.

  Step 702) Unsupervised clustering is performed on the sudden sound section video using the feature amount.

  Step 703) N clusters having a large number of elements are set as hitting ball clusters and stored in the hitting ball event storage unit 80 shown in FIG.

  Next, the video feature extraction processing in step 701 will be described.

  FIG. 13 is a flowchart (No. 1) of the thumbnail image feature amount detection processing according to the embodiment of the present invention. The process shown in the figure is to detect a feature amount using an average value of pixel values when an image is divided into small blocks.

  Step 801) Initialization is performed with x = 0 and y = 0.

  Step 802) With respect to a rectangular area whose upper left is (x, y) and lower right is (x + block_size, y + block_size), an average of pixel values is obtained for each of RGB (Red, Green, Blue), and v [x / Assign to block_size] [y / block_size].

  Step 803) If x + block_size <width-block_size, go to Step 804, otherwise go to Step 805.

  Step 804) The block_size is added to x, and the process proceeds to Step 802.

  Step 805) 0 is substituted for x.

  Step 806) If y + block_size <height-block_size, the process proceeds to Step 807; otherwise, the process ends.

  Step 805) 0 is substituted for x.

  Step 806) If y + block_size <height-block_size, the process proceeds to Step 807; otherwise, the process ends.

  Step 807) The block_size is added to y, and the process proceeds to Step 802.

  The above method uses v as a feature quantity.

  Next, another implementation method of the above step 701 will be described.

  FIG. 14 is a flowchart (part 2) of the thumbnail image feature amount detection process according to the embodiment of the present invention. The process shown in the figure is to detect the feature amount of the video using the direction component of the edge image.

  Step 1001) Initialize with x = 0 and y = 0.

  Step 1002) For the rectangular area whose upper left is (x, y) and lower right is (x + block_size, y + block_size), the x-direction difference of pixel values is obtained for each of RGB, and d_x [x / block_size] [y / Assign to block_size].

  Step 1003) With respect to the sap area of the upper left (x, y) and the lower right (x + block_size, y + block_size), a pixel value y-direction difference is obtained for each of RGB, and d_y [x / block_size] [y Assign to / block_size].

  Step 1004) If x + block_size <width-block_size, go to Step 1006. Otherwise, the process proceeds to step 805.

  Step 1005) Block_size is added to x, and the process proceeds to Step 1002.

  Step 1006) 0 is substituted for x.

  Step 1007) If y + block_size <height-block_size, the process proceeds to step 807, and if not, the process ends.

  Step 1008) Add block_size to y and go to Step 1002.

  In the above method, d_x and d_y are used as feature amounts.

  The image feature amount may be detected using all of v, d_x, and d_y.

  Furthermore, motion features may be used as another method of step 701. In this case, for example, K. Fujii and K. Arakawa, “Video editing based on motion recognition using temporal templates,” Proc, ACM User Interface Software Technology (UIST) 2003, pp. 71-72, 2003 should be used. Is possible. When using the motion feature, the entire video stored in the video storage unit 110 is searched again using the video feature acquired in step 104 as a template between step 104 and step 105 in FIG. Processing is required.

  Next, an evaluation experiment when this example is implemented will be described.

  The content used in the experiment is a major league broadcast video (hereinafter referred to as broadcast video) and a catch ball video (hereinafter referred to as personal shot video) taken by itself.

  Experiments were conducted on three types of broadcast video and two types of personal video. All contents are 44.1kHz mpeg video. Hereinafter, an example of a template MHI (Motion History Image) created by clustering them as a result of detection of a hitting ball sound using sound, and a result of detecting a pitching scene from a video using the template MHI will be described.

  First, an experiment to detect a pitching scene by sound was performed on a catchball image shot with a television broadcast content and a home-use television camera. The results are shown in Table 1.

音による投球シーン検出は球場、撮影環境により、その精度が大幅に変化してしまう。放送映像に関しては、人が聞いて打球音捕球音を何とか聞き取ることができる音響レベルのコンテンツである。そのため、再現率が低めの値になっている。

The accuracy of the pitching scene detection by sound varies greatly depending on the stadium and the shooting environment. The broadcast video is a sound level content that a person can listen to and catch the hitting sound. Therefore, the recall rate is a low value.

  In addition, the catch ball of personal photography is a catch ball while having a conversation, and does not include a large sudden noise other than the conversation. The detection omission is that the catching sound is not included due to a throwing mistake, and the low recall is often due to the detection of the returned ball from the catcher.

  This precision / reproducibility is not a problem because the precision is over 50% for broadcast video as expected, but it is less than 50% for personal video, and the number of elements in the pitch event cluster during clustering. May fall below the number of elements in other clusters.

  In this clustering, if the reproducibility is too low, there is a possibility that a problem may occur when the pitching event detection system based on sound detection is not good. On the other hand, it is necessary to keep the precision rate at 100%. Further, as an implementation problem, the initial cluster element number n increases the amount of calculation by Ο (n2). Considering these situations, n = 3 was used in the subsequent experiments.

  A template is created using a pitch event cluster obtained by clustering among pitch event candidates detected by the hitting sound.

With regard to personal video, since the relevance rate is as low as about 30%, there is a risk that the number of elements of the cluster other than the pitching event will be the largest. However, actually, among the pitching event candidates in the pitching event detection by sound, the movement at that time is various, and it is not put together into one big cluster, and it is empirically personal video The template MHI can be created with sufficient accuracy .

  Table 2 shows the results of pitching scene detection by matching with the template MHI.

また、テンプレートＭＨＩとの距離の時間による推移のグラフを図１５に示す。表２から分かるとおり、再現率が９５．８％、適合率が８５．０％と、音による検出に比べて四度よく検出ができていることがわかる。さらに、適合率について、誤検出の全てがリプレーの投球シーンであった。リプレーはスロー再生しているため理想的には検出しないことが望ましいため不正解としているが、当然ながら非常に困難である。リプレーも正解とするならば、適合率は１００％であり、適合率・再現率ともに高精度であることがわかる。

Further, FIG. 15 shows a graph of the transition of the distance from the template MHI with time. As can be seen from Table 2, the reproducibility is 95.8% and the relevance rate is 85.0%, indicating that the detection can be performed four times better than the detection by sound. Furthermore, with regard to precision, all false detections were replaying scenes. Since replay is slow playback, it is desirable not to detect it ideally, so it is an incorrect answer, but of course it is very difficult. If the replay is also correct, it can be seen that the precision is 100%, and the precision and recall are both highly accurate.

  Further, in the time change of the distance between the template MHI and the personal shot image shown in FIG. 15, all portions where the distance is small are pitching events. As you can see, you can see that the throwing event and other events are well separable and that the template is appropriate. From this, it can be seen that this is extremely useful as baseball video indexing.

  3 or 5 can be implemented by using a computer program to execute a part or all of the functions of the event detection apparatus shown in FIGS. 3 and 5, and the present invention can be realized. It goes without saying that the processing procedures shown in the flowcharts of FIGS. 6 to 9, 11, 13, and 14 and the explanatory text described in the text can be constituted by a computer program and the program can be executed by the computer. In addition, a computer-readable storage medium such as an HDD, an MO, a ROM, a memory card, a program for realizing the function of the computer, or a program for causing the computer to execute the processing procedure. Record it on a CD, DVD, removable disk, etc., save it, or distribute it Rukoto is possible.

  The above program can also be provided through a network such as the Internet or electronic mail.

  The exemplary embodiments and examples of the present invention have been described above, but the present invention is not limited to the above-described embodiments and examples, and various modifications and applications are possible within the scope of the claims. It is.

  The present invention is applicable to a technique for detecting an event from a video.

It is a principle explanatory view of the present invention. It is a principle block diagram of this invention. It is a block diagram of the event detection apparatus in one embodiment of this invention. It is a flowchart of operation | movement of the event detection apparatus in one embodiment of this invention. It is a block diagram of the event detection apparatus of one Example of this invention. It is a flowchart of the event detection process of one Example of this invention. It is a flowchart of the detection of sudden sound of one Example of this invention. It is a flowchart of the calculation method of the power of the acoustic signal of one Example of this invention. It is a flowchart of the suddenness detection process of one Example of this invention. It is an example of how to obtain new_flag F (p (i), flag) according to an embodiment of the present invention. It is a flowchart of the clustering process of the thumbnail image of one Example of this invention. It is an example of the hitting ball event storage part of one Example of this invention. It is a flowchart (the 1) of the image feature-value detection process of the thumbnail of one Example of this invention. It is a flowchart (the 2) of the image feature-value detection process of the thumbnail of one Example of this invention. It is a figure which shows the distance with the template MHI.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Reading means, Reading part 2 Acoustic feature extraction means, Acoustic feature extraction part 3 Event candidate detection means, Event candidate detection part 4 Video feature extraction means, video feature extraction part 5 Clustering means, Clustering part 6 Correct cluster selection means, Correct cluster Selection unit 7 Template creation unit 8 Template use re-search unit 10 Video reading unit 20 Sudden sound segment detection unit 40 Sudden sound segment video acquisition unit 50 Sudden sound segment video clustering unit 70 Template creation unit 80 Template re-search unit 110 Video storage means , Video storage unit 120 event storage means, event storage unit, hitting ball event storage unit

Claims (13)

In the event detection method for detecting a specific event in the video,
A video reading step for reading video from the video storage means;
An acoustic feature detection step of detecting an acoustic feature in the read video;
An event candidate detection step for detecting an event candidate based on the video section in which the acoustic feature is detected;
A video feature extraction step of extracting video features in the video segment of the event candidate;
A clustering step for performing unsupervised clustering of video candidates based on the video features;
A correct cluster selection step of storing the upper cluster having a large number of elements in the storage means as a correct cluster;
An event detection method characterized by: After the correct cluster selection step,
A template image acquisition step of acquiring a template image from the clustering result;
An event re-search step for re-searching the entire video of the video storage means by the template image;
The event detection method according to claim 1, further comprising: In the acoustic feature detection step,
The event detection method according to claim 1, wherein a portion having a sudden sound characteristic in the sound power transition is extracted as the acoustic feature. In the video feature extraction step,
The event detection method according to claim 1, wherein an RGB value of an image in the event candidate section is used as the video feature. In the video feature extraction step,
The event detection method according to claim 4, wherein a direction component of an edge image is used as the video feature. In the video feature extraction step,
The event detection method according to claim 4, wherein a motion feature is used as the video feature. An event detection device for detecting a specific event from a video,
Video storage means for storing video;
Video reading means for reading video from the video storage means;
Acoustic feature detecting means for detecting acoustic features in the read video;
Event candidate detection means for detecting an event candidate based on the video section in which the acoustic feature is detected;
Video feature extraction means for extracting video features in the event candidate video section;
Clustering means for performing unsupervised clustering of video candidates based on the video features;
Correct cluster selection means for storing the upper cluster having a large number of elements in the storage means as the correct cluster;
An event detection apparatus comprising: Template image acquisition means for acquiring a template image from the clustering result;
Event re-search means for re-searching the entire video of the video storage means by the template image;
The event detection apparatus according to claim 7, further comprising: The acoustic feature detection means includes
The event detection device according to claim 7, wherein a portion having a sudden sound characteristic in the sound power transition is extracted as the acoustic feature. The video feature extraction means includes
The event detection apparatus according to claim 7, wherein an RGB value of an image in the event candidate section is used as the video feature. The video feature extraction means includes
The event detection apparatus according to claim 10, wherein a direction component of an edge image is used as the video feature. The video feature extraction means includes
The event detection apparatus according to claim 10, wherein a motion feature is used as the video feature. An event detection program for detecting a specific event from a video,
An event detection program causing a computer to execute a method for realizing the event detection method according to claim 1.

Images