JP2005332206A - Video event discrimination device, program thereof, learning data generation device for video event discrimination, and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video event discrimination device which discriminates an event occurring within a video, from each scene of the video without using additional information. <P>SOLUTION: A video event discrimination device 2 is provided with; a scene division means 12 for dividing an inputted video into scenes; a feature quantity extraction means 14 for extracting video feature quantities from a plurality of frame images included in scenes; a feature quantity digitizing means 21 for converting video feature quantities to numerical data by referring to a feature quantity classification database wherein representative values of similar video feature quantities are preliminarily associated with numerical data for classifying the representative values; and an event specifying means 22 for specifying a classification of an event corresponding to a data string of the numerical data converted by the feature quantity digitizing means 21 by referring to an event database wherein classifications of events are preliminarily associated with scene digitized strings resulting from representing a plurality of consecutive scenes by data strings of numerical data. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video event discriminating apparatus and program for discriminating an event occurring in a video, and a video event discriminating learning data generating apparatus and program thereof.

  In recent years, various event discriminating techniques have been proposed for discriminating events occurring in video from video such as broadcast programs. For example, as a first event discrimination technique, the occurrence of a video event is detected and the type of the event is discriminated by detecting that character information displayed at a specific position on the screen is changed. A technique is disclosed (see Patent Document 1). This technology uses the fact that character information indicating the progress of the game, such as “inning”, “score”, “number of outs”, etc., is displayed at a specific position on the screen in a baseball broadcast video. By detecting this change, it is possible to discriminate events such as the turning point of inning and the acquisition of scores.

For example, as a second event discrimination technique, a technique for discriminating an event in a sports broadcast video by relay data (score information or the like) distributed via the Internet is disclosed (see Patent Document 2). In this technique, baseball broadcast videos are sequentially recorded, and the recorded video is played back a predetermined time (for example, 10 minutes) at the stage where the scoring scene is recognized by the score information distributed via the Internet. As described above, in the second event discrimination technique, a scoring scene that becomes a baseball relay event is discriminated based on relay data linked to a video.
JP 2000-132563 A (paragraphs 0048 to 0049, FIG. 5) JP 2003-174609 A (paragraphs 0014 to 0026, FIGS. 1 to 4)

In the first event discriminating technique described above, the event is discriminated based on the unique character information on the video, so that the event cannot be discriminated unless the character information is presented on the screen. is there.
Further, in the second event discrimination technique, means for acquiring information other than video in order to discriminate events occurring in the video by acquiring information (relay data) linked to the video from the Internet or the like. There is a problem that the apparatus configuration becomes complicated. Furthermore, in the second event determination technology, in the video broadcast in real time, the event can be determined by acquiring the corresponding relay data, but from the video accumulated by recording or the like, There is also a problem that the event cannot be determined.

  As described above, in the first and second event discriminating technologies described above, since the event in the video is discriminated based on the information (character information, relay data) added to the video, the added information is In a situation where it cannot be obtained, the event cannot be determined. Therefore, there is an increasing demand for technology development that can discriminate events from video scenes themselves.

  The present invention has been made to solve the above-described problems, and a video event discriminating apparatus and a program for discriminating an event occurring in the video from each scene of the video without using additional information, It is another object of the present invention to provide a video event discrimination learning data generation apparatus and a program thereof.

  The present invention has been developed to achieve the above object, and first, the video event determination device according to claim 1 is configured so that the video in-video is based on the video feature amount in each scene of the input video. A video event discriminating apparatus for discriminating the type of an event occurring in a feature class database storage means, an event database storage means, a scene dividing means, a feature quantity extraction means, a feature quantity quantification means, an event And a specific means.

  According to such a configuration, the video event determination device detects a point (scene change) where the screen configuration of the input video is largely switched by the scene dividing unit, and divides the video for each scene. Then, the video event discriminating apparatus extracts the video feature amount indicating the video feature in the scene from the plurality of frame images included in the scene divided by the scene dividing unit by the feature amount extracting unit.

  Further, the video event discriminating apparatus converts the video feature amount extracted by the feature amount extraction unit by the feature amount digitizing unit into numerical data (cluster value) for specifying a class in which the video feature amount is classified (clustered) in advance. Convert. Note that the numerical data obtained by classifying the video feature quantity is stored in advance in the feature quantity classification database storage unit as a feature quantity classification database associated with the video feature quantity. By referring to the feature quantity classification database, the feature quantity digitizing means can represent the video scene with simplified numeric data.

  Then, the video event discriminating device specifies the type of event corresponding to the data string of the numerical data converted by the feature value digitizing means by the event specifying means. The event type is stored in advance in the event database storage unit as an event database in which the event type is associated with a scene digitization sequence in which a plurality of continuous scenes are represented by a data sequence of numerical data. By referring to this event database, the event specifying means can specify the type of event from the scene digitization sequence.

  According to a second aspect of the present invention, there is provided the video event determination device according to the first aspect, wherein the video event determination device includes a reference image storage unit, and the scene division unit includes an event start detection unit.

  According to this configuration, the video event determination device stores a reference image (reference image) indicating the start of an event in the reference image storage unit in advance. Then, the event start detecting means in the scene dividing means detects a frame image similar to the reference image in the input video frame image as the start of the event. As a result, in addition to scene switching, a frame image in which an event to be discriminated starts is set as the start of the scene.

  As the reference image, for example, in a baseball broadcast video, when it is desired to determine an operation after the batter is standing at the bat as an event, the reference image is an image when the batter is standing at the bat. By storing this reference image in the reference image storage means, the event start detection means can determine that the scene is a different scene from the scene where the batter stands in the same scene.

  Furthermore, the video event determination program according to claim 3, wherein the computer uses a scene dividing unit to determine the type of event occurring in the video based on the video feature amount in each scene of the input video. , Feature quantity extraction means, feature quantity quantification means, and event identification means.

  According to this configuration, the video event determination program detects a point (scene change) where the screen configuration of the input video is largely switched by the scene dividing unit, and divides the video for each scene. Then, the video event determination program extracts the video feature amount indicating the video feature in the scene from the plurality of frame images included in the scene divided by the scene dividing unit by the feature amount extracting unit.

  Furthermore, the video event determination program refers to a feature quantity classification database in which representative values of similar video feature quantities and numerical data for classifying the representative values are associated in advance by the feature quantity quantification unit. The video feature quantity extracted by the quantity extraction means is converted into numerical data (cluster value) that identifies the classified (clustered) class.

  Then, the video event determination program refers to an event database in which event types are associated in advance with event types and scene digitization sequences in which a plurality of continuous scenes are represented by a data sequence of numerical data. The event type corresponding to the data string of the numerical data converted by the feature value digitizing means is specified.

  According to a fourth aspect of the present invention, there is provided a video event discriminating learning data generating apparatus for generating a feature quantity classification database and an event database, which are learning data used in the video event discriminating apparatus according to the first or second aspect. The event discrimination learning data generating apparatus is configured to include a scene dividing unit, a feature amount extracting unit, a feature amount classifying unit, a scene digitizing unit, a scene video reproducing unit, and an event setting unit.

  According to such a configuration, the video event determination learning data generation apparatus detects a point (scene change) where the screen configuration of the input video is largely switched by the scene dividing unit, and divides the video for each scene. Then, the video event determination learning data generation apparatus extracts a video feature amount indicating a video feature in the scene from a plurality of frame images included in the scene divided by the scene division unit by the feature amount extraction unit.

  Furthermore, the learning data generation device for video event discrimination classifies the video feature amount extracted by the feature amount extraction unit by the feature amount classification unit in association with numerical data for each representative value of similar video feature amounts. . The representative value of the similar video feature amount (representative video feature amount) represents all the video feature amounts included in the classified class when the video feature amount is classified. The average value of all the video feature quantities belonging to the class or the video feature quantity closest to the average value can be set. As described above, the feature amount classification unit generates a feature amount classification database used in the video event determination device by associating the representative value of the similar video feature amount with the numerical data indicating the class.

  The learning data generation device for video event determination includes a frame image number for identifying a frame image included in the scene and a video feature amount classified by the feature amount classification unit for each scene by the scene digitizing unit. Is associated with the numerical data. As a result, which scene is represented by which numerical data is associated.

  The video event determination learning data generation apparatus reproduces a scene based on the frame image number of each scene associated by the scene digitizing means by the scene video reproducing means. Then, the video event determination learning data generation device receives event identification information indicating the type of event for a plurality of consecutive reproduced scenes by the event setting means, and the event identification information Is associated with a scene digitization sequence that is a data sequence of numerical data of video feature values corresponding to the scene. As described above, the event setting unit generates an event database used in the video event determination device by associating the event type (event identification information) with the scene digitization sequence.

  Further, the video event determination learning data generation device according to claim 5 is the video event determination learning data generation device according to claim 4, further comprising reference image storage means, wherein the scene division means detects event start. It was set as the structure provided with a means.

  According to this configuration, the video event determination learning data generation apparatus stores in advance the reference image (reference image) indicating the start of the event in the reference image storage unit. Then, the event start detecting means in the scene dividing means detects a frame image similar to the reference image in the input video frame image as the start of the event. As a result, in addition to scene switching, a frame image in which an event to be determined is started is set as the start of the scene.

  According to a sixth aspect of the present invention, there is provided a video event determination learning data generation program for generating a feature amount classification database and an event database, which are learning data used in the video event determination device according to the first or second aspect. In addition, the computer is configured to function as a scene dividing unit, a feature amount extracting unit, a feature amount classifying unit, a scene digitizing unit, a scene video reproducing unit, and an event setting unit.

  According to such a configuration, the learning data generation program for video event determination detects a point (scene change) where the screen configuration of the input video is largely switched by the scene dividing unit, and divides the video for each scene. Then, the learning data generation program for video event determination extracts the video feature amount indicating the video feature in the scene from the plurality of frame images included in the scene divided by the scene division unit by the feature amount extraction unit.

Further, the learning data generation program for video event discrimination classifies the video feature amount extracted by the feature amount extraction unit by the feature amount classification unit in association with numerical data for each representative value of similar video feature amounts. Thus, a feature amount classification database used in the video event determination device is generated.
Then, the learning data generation program for determining the video event discriminates the frame image number for identifying the frame image included in the scene for each scene by the scene digitizing unit and the video feature amount classified by the feature amount classifying unit. Is associated with the numerical data.

  The video event discrimination learning data generation program reproduces the scene based on the frame image number of each scene associated by the scene digitizing means by the scene video reproducing means. Then, the video event discrimination learning data generation program receives event identification information indicating a type of event for a plurality of reproduced continuous scenes by the event setting means, and a plurality of event identification information, An event database to be used in the video event discriminating apparatus is generated by associating with a scene digitization sequence that is a data sequence of numeric data of video feature values corresponding to the scene.

  According to the invention described in claim 1 or claim 3, a feature amount for each scene is extracted from a video, and the video is represented as a data string of numerical data by numerical data indicating a class in which the feature amount is classified. By referring to an event database in which a data string of numerical data is associated with an event in a video in advance, it is possible to determine the type of event that occurs in the video. In addition, according to the present invention, since the type of event that occurs in the video is determined by the simplified data string of the numerical data obtained by extracting the features of the video, the event can be determined at high speed, and information other than the video ( There is no need to use additional information such as character information. As a result, the present invention can determine an event whether it is a real-time video or an accumulated video.

  According to the second aspect of the present invention, when a frame image similar to a reference image indicating the start of an event is input as an input video, it is possible to divide the scene as a video scene change Become. That is, according to the present invention, the first frame image of a scene where an event occurs can be specified in the input video. Therefore, according to the present invention, a data string of numerical data corresponding to the feature amount of the video starting from the frame image where the event is surely started is generated, and the accuracy of event determination can be improved. .

  According to the invention described in claim 4 or claim 6, the feature quantity classification database can be generated by extracting the video feature quantity for each scene from the video and classifying (clustering) the video feature quantity. it can. Further, according to the present invention, it is possible to generate an event database in which an event type constituted by a plurality of scenes is associated with a data string obtained by connecting data strings of numerical data for each scene. By using the feature amount classification database and the event database, the video event determination device can express a video as a data string of clustered numerical data, and determines an event based on the data string. It becomes possible.

  According to the fifth aspect of the present invention, the learning data generation program for video event determination is configured to divide a scene as a video scene change at a stage where a frame image similar to a reference image indicating the start of an event is input. It becomes possible to do. That is, according to the present invention, the first frame image of a scene where an event occurs can be specified in the input video. Therefore, numerical sequence data is generated based on the feature amount of the video starting from the frame image where the event is surely started, and the video event discrimination device can improve the accuracy of event discrimination.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Outline of video event discrimination method]
First, with reference to FIG. 1, the outline | summary is demonstrated about the method of discriminating the event which generate | occur | produces in an image | video from an image | video in the video event determination apparatus based on this invention. FIG. 1 is an explanatory diagram for explaining the outline of the video event determination method. Here, an event refers to a series of scenes having a certain meaning in a video, for example, a scene where a “home run”, “double strike”, etc. occur in a baseball broadcast video. FIG. 1 shows an example in which “double strike” is determined as an event from the baseball broadcast video.

Here, the video V is composed of scenes V _{1 to} V ₄ as scenes in which the composition of the camera is switched. The scene V ₁ is a “scene where the pitcher throws”, and the V ₂ is a ball where the batter hits the outside field. “Scene” and V ₃ indicate “scene where the runner went around the base 1”, and V ₄ shows “scene where the runner stopped on the base 2”.

Then, the video event determination method extracts the video feature amount Vc from each of the scenes V _{1 to} V ₄ . Here, the video feature amount Vc is shown as information of a rectangular area in which the video V is simplified for each scene. For example, the scene V ₁ is simplified to information of seven rectangular areas (rectangle 1 to rectangle 7).

In the video event discrimination method, based on the feature quantity classification database 10a in which similar video feature quantities Vc are classified (clustered) in advance, the video feature quantity Vc for each scene is represented as a cluster number cluster C _mn (1 ≦ C _{It is} digitized as _mn ≦ N). Furthermore, the video event discrimination method is based on the event database 11a in which a video event type (identification information) and a scene digitization sequence in which a plurality of continuous scenes are represented by a data sequence of numerical data are associated in advance. It is determined which event is a plurality of scenes. Here, the event is “double hit”.
In this way, the video event discrimination method expresses a video as a data string of numerical data simplified based on the video feature amount, and discriminates an event (video event) occurring in the video based on the data string. To do.

  Here, as an event of a video, an event occurring in a baseball broadcast video (double hit) is illustrated, but the present invention is not limited to an event of a baseball relay video. The present invention can discriminate an event if it is a video whose camera work and composition are determined by the event. For example, in a cooking program video, etc., if camera work and composition are determined for the work process, it is possible to discriminate "scenes that cut cabbage", "scenes that grill fish", etc. .

  As shown in FIG. 1, the video event determination method refers to learning data (feature amount classification database 10a and event database 11a) learned in advance, and performs video event determination. A learning data generation apparatus that generates learning data will be described, and subsequently, a video event determination apparatus that implements the video event determination method illustrated in FIG. 1 will be described sequentially.

[Configuration of learning data generator]
First, the configuration of a learning data generation device (video event determination learning data generation device) according to the present invention will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the learning data generation apparatus.

  As illustrated in FIG. 2, the learning data generation device 1 numerically represents a feature amount classification database 10 a in which similar video feature amounts are classified from an externally input video, and a plurality of scenes continuous with the types of video events. An event database 11a in which a scene digitization sequence represented by a data sequence of data is associated is generated as learning data.

  Here, the learning data generation device 1 includes a feature quantity classification DB storage means 10, an event DB storage means 11, a scene division means 12, a reference image storage means 13, a feature quantity extraction means 14, and a video feature quantity storage. Means 15, feature amount classification means 16, scene digitization means 17, scene classification DB storage means 18, scene video reproduction means 19, and event setting means 20 are provided.

  The feature quantity classification DB (database) storage means 10 stores a feature quantity classification database 10a in which video feature quantities for each scene are classified (clustered) for each similar video feature quantity. Storage means.

The event DB (database) storage means 11 stores an event database 11a in which event types are associated with scene digitization sequences in which a plurality of scenes are represented by a data sequence of numerical data. This is a general storage means.
The feature quantity classification database 10a and the event database 11a are generated in the learning data generation apparatus 1.

  The scene dividing means 12 inputs an image from outside and divides the image for each scene. The scene dividing unit 12 detects a point (scene change) at which the screen configuration of the video is largely switched, and divides the video at each switching point. This scene change can be detected by an existing method. For example, the scene dividing unit 12 calculates a numerical vector (for example, each average value of RGB) based on each color feature from the time-series frame images constituting the video, and calculates the numerical vector of the preceding and succeeding frame images. If the sum of absolute values of the differences is larger than a predetermined threshold value, it is determined that there is no continuity between the frame images and it is considered that a scene change has occurred.

  The scene dividing unit 12 sequentially outputs a frame image to the feature amount extracting unit 14 for each scene obtained by dividing the video. Note that the scene dividing unit 12 detects (divides) the next scene when the numerical value data of one scene is generated by the scene digitizing unit 17 described later. Further, here, the scene dividing unit 12 includes an event start detecting unit 12a.

The event start detection means 12a detects a scene switching start point (switched image) from the input video. The event start detection means 12a compares the frame image with the reference image 13a stored in the reference image storage means 13 to detect that a frame image similar to the reference image 13a has been input. A frame image similar to the image 13a is regarded as a scene switching start point. This similarity determination is performed, for example, by comparing the sum of absolute values of differences between the reference image 13a and the frame image with a predetermined threshold value.
By making the reference image 13a an image that is the start of an event, for example, the scene dividing unit 12 can reliably divide the scene from the first frame image in which the event occurs.

  The reference image storage unit 13 stores a reference image 13a that is the start of an event, and is a general storage unit such as a hard disk. For example, when generating learning data for discriminating an event from a video of a baseball broadcast, the reference image 13a is an image when a batter is standing at a bat or an image when a pitcher performs a pitch, The scene dividing means 12 can detect the start of various events in baseball (for example, home run, strikeout, etc.).

The feature quantity extraction unit 14 extracts a video feature quantity from the frame image constituting the scene for each scene divided by the scene division unit 12. As this video feature amount, a general video feature amount can be used. For example, it is a numerical vector of RGB average values in all frame images constituting the scene. Note that the feature amount extraction unit 14 does not target all frame images in the scene, but selects a frame image at a predetermined sampling interval and extracts a video feature amount from the selected frame image. It is good.
Note that the video feature quantity extracted here is stored in the video feature quantity storage unit 15. In addition, the feature quantity extraction unit 14 notifies the feature quantity classification unit 16 that the video feature quantity has been extracted at the stage where the video feature quantity is stored in the video feature quantity storage unit 15.

  In the present embodiment, the feature quantity extraction means 14 uses the “video feature information generation method, video feature information generation apparatus, and video feature information generation program” (Japanese Patent Application No. 2003-73548) filed by the applicant of the present application. The video feature amount of the scene is expressed by a rough rectangular area and the image feature amount and motion of the rectangular area. Therefore, here, the feature quantity extraction means 14 includes a node tracking means 14a, a node classification means 14b, a cluster image feature quantity generation means 14c, and a scene feature quantity generation means 14d.

The node tracking unit 14a sets points (nodes) serving as a reference for extracting the features of the frame image in a grid pattern at predetermined intervals in the first frame image of the scene, and features of the neighboring image region of the node The nodes are tracked for each frame image based on the quantity.
Here, with reference to FIG. 7 (refer to FIG. 2 as appropriate), the tracking of the node in the node tracking means 14a will be described. FIG. 7 is a diagram visualizing the tracking of a node in the node tracking means. FIG. 7A is a diagram showing a state in which the node is set on the frame image, and FIG. 7B is a diagram showing a state in which the node is tracked on the frame image. FIG.

  As shown in FIG. 7 (a), the node tracking means 14a sets the nodes PT in a grid pattern with Npx horizontal and Npy vertical (Npx and Npy are preset) in the first frame image of the scene. Then, the node tracking unit 14a calculates an image feature amount from a neighboring region (neighboring region image: square region of Rfv pixels × Rfv pixels) centered on each node PT, and associates it with each node PT. The image feature amount may be a general feature amount in the field of image processing. For example, the average value of each RGB color component, the distribution of the edge amount when the image is edged, and the fractal indicating the complexity of the image Dimension etc. can be used.

In FIG. 7A, for convenience of explanation, the node PT is illustrated on the frame image, but the node PT merely indicates a position corresponding to a grid-like point on the frame image. .
Then, as shown in FIG. 7 (b), the node tracking means 14a determines in advance that the image feature quantity in the region near the node PT (marked with ● in the figure) in the previous frame image is the current frame image. The node is moved to a position PT _B (x mark in the figure) where the difference is the smallest below the threshold value. As a result, a region where the image feature amount approximates is tracked throughout the scene.
Returning to FIG. 2, the description will be continued.

  The node classification means 14b classifies (clusters) the nodes in the frame image for each frame image based on the position of the node and the image feature amount of the neighboring region. The node classifying unit 14b classifies the nodes that are approximate to the image feature amount of the neighboring region at each node as the same class (cluster). However, with only the image feature amount, as shown in FIG. 8A, nodes separated from each other are classified as the same cluster CL1, and therefore the node classification means 14b is arranged as shown in FIG. 8B. Then, nodes that are more than a predetermined distance away from any node of the same cluster are separated and classified as separate clusters (CL1 and CL2).

  The cluster image feature value generating unit 14c includes the nodes tracked by the node tracking unit 14a and the nodes of the same cluster classified by the node classifying unit 14b from the first frame image to the final frame image of the scene by the node tracking unit 14a. The image feature amount (cluster image feature amount) of the cluster is generated at the stage where the process is tracked to the point. This cluster image feature amount is, for example, an average value of image feature amounts of each node from the first frame image to the last frame image of the scene in the same cluster. It should be noted that an image that most closely approximates the average value of the image feature values among the adjacent region images of the nodes may be used as one of the image feature values as the representative texture image of the cluster.

  The scene feature quantity generation unit 14d generates an image feature quantity for each cluster in the entire scene as a feature quantity (video feature quantity) of the scene. The scene feature quantity generation unit 14d generates coordinate information of a rectangular area indicating a cluster area, the cluster image feature quantity generated by the cluster image feature quantity generation unit 14c, and the movement of the cluster as a scene feature quantity. . The rectangular area indicating the cluster area indicates the maximum area including the coordinates of the nodes included in the same cluster in the entire scene. The cluster motion indicates a motion vector of the position centroid of the cluster from the first frame image to the last frame of the scene.

Here, FIG. 9 shows an example of video feature value data generated by the feature value extraction means 14. As shown in FIG. 9, for each scene, the first frame number Ns, the last frame number Ne, and the coordinate information {(x0, y0), (x1, y1), (x2, y2), (x3, y3)}, image feature quantity {(f (0), f (1), f (2),..., f (N-1)}, and cluster with x and y components {vx, vy} of the motion vector This is one piece of information.
As a result, the feature quantity extraction means 14 (FIG. 2) simplifies the scene of the video V with a plurality of rectangular areas R (R ₁ , R ₂ , R ₃ ), as shown in FIG. Video feature quantities are extracted as coordinate information (position, size), image feature quantities, and motion vectors.
Returning to FIG. 2, the description will be continued.

  The video feature quantity storage means 15 stores the video feature quantity 15a for each scene extracted by the feature quantity extraction means 14, and is a storage means such as a hard disk. The video feature quantity storage means 15 functions as a buffer for temporarily storing the video feature quantity 15a, and the numerical value string data of all scenes in the input video is generated by the scene digitizing means 17 described later. It is deleted at the stage.

  The feature quantity classifying means 16 classifies (clusters) the video feature quantities 15a extracted by the feature quantity extracting means 14 and stored in the video feature quantity storage means 15 for each similar video feature quantity, and the video feature quantities 15a are classified. It is associated with the value (cluster value) of the classified class (cluster). When the feature quantity classifying unit 16 is notified by the feature quantity extracting unit 14 that the video feature quantity 15a has been extracted, the feature quantity classifying unit 16 compares the plurality of video feature quantities 15a stored in the video feature quantity storage unit 15 with a difference. A cluster that is equal to or less than a predetermined value (threshold value) is defined as one cluster. Further, the feature quantity classifying unit 16 sets the average value of the video feature quantities included in the same cluster as the video feature quantity (representative video feature quantity) representing the cluster, and the feature quantity classification database 10a associated with the cluster value. And is stored in the feature amount classification DB storage unit 10.

It should be noted that the feature quantity classifying unit 16 determines that each of the video feature quantities 15a is a feature vector composed of a plurality of feature quantities (for example, coordinate information, image feature quantities, and motion vectors as shown in FIG. 9). An average value is calculated for each feature amount and used as a representative video feature amount.
Further, as shown in the example of the structure diagram of the feature quantity classification database in FIG. 11, the feature quantity classification database 10a includes cluster values C (C ₁ , C ₂ ,...) And representative video feature quantities CV (CV ₁ , CV). ₂ ,...) In a one-to-one correspondence.

  The scene digitizing means 17 associates a scene with a numerical value (cluster value) by converting the video scene into a cluster value classified by the feature quantity classifying means 16 based on the video feature quantity of the scene. is there. Here, since the scene is represented by a plurality of rectangular areas, the scene digitizing means 17 converts the scene into a data string of a plurality of numerical values (cluster values). The scene digitizing means 17 generates the scene classification database 18a by associating the digitized data with the frame number of each scene, and stores it in the scene classification DB storage means 18.

  As shown in the example of the structure diagram of the scene classification database in FIG. 12, the scene classification database 18a includes a frame number Fn (first number Fs-final number Fe), a scene number Sn that is a serial number for specifying a scene, It is a database in which one or more cluster values C are associated with each other.

  The scene classification DB (database) storage means 18 stores the scene classification database 18a generated by the scene digitizing means 17, and is a storage means such as a hard disk. The scene classification database 18a stored in the scene classification DB storage means 18 is referred to by a scene video reproduction means 19 and an event setting means 20 described later.

The scene video playback means 19 plays back video for each scene by referring to the scene classification database 18a. The scene video reproduction means 19 sequentially reproduces the images of the corresponding frame numbers Fn in the order of the scene numbers Sn in the scene classification database 18a shown in FIG. 12, and displays the scene reproduction video on a display device (not shown). Thus, a video for each scene is presented to the operator who operates the learning data generation apparatus 1.
The scene video playback means 19 displays on the screen of the display device a slide bar that can specify the specified position on the entire time axis of the video, and the operator appropriately slides by means of input means such as a mouse (not shown). By operating the bar, the corresponding scene may be reproduced and an event occurring in the video may be confirmed as a reproduced video.

  The event setting means 20 is instructed by the operator via an input means (not shown) as a plurality of continuous scenes as events, thereby converting the event types and scene numerical values representing the plurality of scenes into numerical string data. Corresponds to a column. Note that identification information (event name or the like) for identifying an event is input from an input unit (not shown). The scene corresponding to the event is directly input with the scene number, or the event setting means 20 searches the scene classification database 18a for the scene number according to the position of the slide bar.

  As a result, the event setting means 20 can associate the event type with the scene (scene number). Therefore, the event setting means 20 associates a scene digitization sequence obtained by connecting a cluster value data sequence associated with a scene number in the scene classification database 18a for a plurality of scenes with event identification information (event name, etc.). Thus, the event database 11a is generated and stored in the event DB storage means 11.

As shown in the example of the configuration diagram of the event database in FIG. 13, the event database 11a is a database in which the event identification information Ek is associated with the scene digitization sequence Sd. In the example of FIG. 13, the event value “home run” (event identification information Ek) includes the cluster value (C ₁₁ , C ₁₂ , C ₁₃ ,...) And the scene number Sn _{2 of} the scene number Sn ₁ shown in FIG. Are associated with each other (C ₂₁ , C ₂₂ , C ₂₃ ,...),.

  As described above, the learning data generation device 1 has a feature amount classification database 10a in which similar video feature amounts are classified into learning data for discriminating an event (video event) from an input video, It is possible to generate the event database 11a in which the type of video event and the scene digitization sequence in which a plurality of continuous scenes are represented by a data sequence of numerical data are associated with each other.

  Note that the learning data generation apparatus 1 can omit the configurations of the event start detection unit 12a and the reference image storage unit 13 in the scene division unit 12. However, it is preferable to provide the event start detection unit 12a and the reference image storage unit 13 because numerical data is generated from the beginning of the event and the event can be appropriately converted into a data string of numerical data.

  The learning data generation device 1 can be realized by causing a general computer to execute a program and operating an arithmetic device or a storage device in the computer. The program (learning data generation program for video event determination) can be distributed via a communication line, or can be written and distributed on a recording medium such as a CD-ROM.

[Operation of learning data generator]
Next, with reference to FIG. 3 and FIG. 4, the operation of the learning data generation device (video event determination learning data generation device) according to the present invention will be described. Here, the operation of the learning data generation device will be described separately for an operation for generating a feature amount classification database and an operation for generating an event database. FIG. 3 is a flowchart illustrating an operation in which the learning data generation device generates a feature amount classification database. FIG. 4 is a flowchart illustrating an operation in which the learning data generation apparatus generates an event database.

(Feature classification database generation operation)
First, referring to FIG. 3 (refer to FIG. 2 as appropriate), an operation in which the learning data generation device 1 generates the feature amount classification database 10a will be described.
First, the learning data generation device 1 inputs a video in units of frame images by the scene dividing unit 12 (step S1). Then, the learning data generation device 1 determines whether the input original frame image is similar to the reference image 13a stored in the reference image storage unit 13 by the event start detection unit 12a (step S2). .

  Here, when the original frame image and the reference image 13a are not similar (No in step S2), the scene dividing unit 12 compares the original frame image with the previous frame image input previously in the time direction. The continuity between the frame images is determined (step S3). When the scene dividing unit 12 determines that the frame images have continuity (Yes in step S3), the scene dividing unit 12 stores the frame images in a storage unit (not shown), returns to step S1, and returns to the next frame. Enter an image.

When the original frame image and the reference image 13a are similar (Yes in step S2) or when the frame image is not continuous (No in step S3), the scene is switched in the original frame image ( It is regarded as a scene change), and the original frame image is set as a scene change image (step S4).
Here, the scene dividing unit 12 determines whether or not the original frame image is the first switching image (step S5). If the original frame image is the first switching image (Yes in step S5), the scene dividing unit 12 displays the original frame image. The image is stored in storage means not shown, and the process returns to step S1 to input the next frame image.

  If the original frame image is not the first switching image (No in step S5), the learning data generating device 1 uses the feature amount extraction unit 14 to change the scene from the immediately preceding scene switching image to the immediately preceding frame image. The video feature amount is extracted from (step S6). In the learning data generating apparatus 1, the node tracking unit 14a tracks a node predetermined in the first frame image of the scene for each frame image based on the feature amount of the image area near the node. In addition, the node classification unit 14b classifies (clusters) the nodes in the frame image for each frame image based on the position of the node and the image feature amount of the neighboring region. Then, the cluster image feature value generating unit 14c determines each node tracked by the node tracking unit 14a and the node of the same cluster classified by the node classifying unit 14b from the first frame image of the scene by the node tracking unit 14a. When the frame image is tracked, an image feature amount (cluster image feature amount) of the cluster is generated. Then, the scene feature value generation unit 14d generates an image feature value for each cluster in the entire scene as a feature value (video feature value) of the scene.

  Subsequently, the learning data generation device 1 classifies (clusters) the video feature quantities extracted in step S6 by the feature quantity classification unit 16 for each similar video feature quantity, and classifies the video feature quantities into the classified classes ( The feature quantity classification database 10a is generated in association with the cluster value (cluster value) (step S7).

Then, the learning data generation device 1 determines whether or not the input video has ended (step S8). If it has not ended (No in step S8), the learning data generation device 1 returns to step S1 and continues the operation. If the video is finished (Yes in step S8), the operation is finished.
Although not shown in the figure, it is assumed that the process proceeds to step S6 when it becomes impossible to input a video in step S1. As a result, the video feature amount of the final scene of the video is extracted.
As described above, the learning data generation device 1 can generate the feature quantity classification database 10a obtained by clustering the video feature quantities for each scene from the input video.

(Event database generation operation)
Next, with reference to FIG. 4 (refer to FIG. 2 as appropriate), an operation in which the learning data generation device 1 generates the event database 11a will be described. Note that the operation from step S11 to step S16 in FIG. 4 is the same as the operation from step S1 to step S6 described with reference to FIG. 3, and thus the description thereof will be omitted, and the operation after step S17 will be described.

After the extraction of the video feature amount in step S16, the learning data generating apparatus 1 uses the scene digitizing means 17 to add a frame number (start number-final number) and a data string of a plurality of numerical values (cluster values) to the scene number. Is associated with the scene classification database 18a (step S17).
Then, the learning data generation device 1 determines whether or not the input video has ended (step S18), and if it has not ended (No in step S18), it returns to step S11 and continues the operation.

  On the other hand, when the video is completed (Yes in step S18), the learning data generating device 1 plays the video for each scene by the scene video playback means 19 (step S19). The learning data generation device 1 then instructs the event setting unit 20 to specify a plurality of continuous scenes as events via an input unit (not shown) from the operator, thereby setting the event type to a scene of a plurality of scenes. The event database 11a is generated in association with the digitized string (step S20).

Then, the learning data generation device 1 determines whether or not an end instruction for event association is input from the operator (step S21), and ends the operation when the end is instructed (Yes in step S21). While not instructed (No in step S21), the process returns to step S19 to continue the operation.
As described above, the learning data generation device 1 generates the event database 11a in which the data sequence of the cluster value of the video feature amount is associated with the event in which a plurality of scenes are continuous from the input video. Can do.

[Configuration of video event discriminator]
Next, the configuration of the video event determination device according to the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing the configuration of the video event discriminating apparatus.
As shown in FIG. 5, the video event discriminating apparatus 2 discriminates the type of event from the video inputted from the outside. Here, the video event discriminating apparatus 2 includes a feature amount classification DB storage unit 10, an event DB storage unit 11, a scene division unit 12, a reference image storage unit 13, a feature amount extraction unit 14, and a video feature amount storage. Means 15, feature value digitizing means 21, and event specifying means 22 are provided.

  Here, the configuration other than the feature value digitizing means 21 and the event specifying means 22 is the same as the configuration of the learning data generating apparatus 1 described with reference to FIG. In the learning data generation device 1 (FIG. 2), when the event start detection unit 12a and the reference image storage unit 13 are omitted from the configuration, the video event determination device 2 is also omitted from the configuration.

  The feature quantity classification database 10a (see FIG. 11) stored in the feature quantity classification DB storage means 10 and the event database 11a (see FIG. 13) stored in the event DB storage means 11 are used as learning data. It is generated in advance by the learning data generation device 1.

  The feature quantity digitizing means 21 converts the video feature quantity extracted by the feature quantity extracting means 14 into a numeric value (cluster value) obtained by classifying the video feature quantity based on the feature quantity classification database 10a. The feature quantity digitizing means 21 is notified of the fact that the video feature quantity has been extracted from the feature quantity extracting means 14, and the video feature quantity 15a stored in the video feature quantity storage means 15 is shown in FIG. The cluster value C having the shortest data distance from the representative video feature quantity CV in the feature quantity classification database 10a is set as the cluster value of the video feature quantity 15a. The cluster value converted here is output to the event specifying means 22.

  The event specifying means 22 specifies which event the data string of cluster values sequentially output from the feature value digitizing means 21 corresponds to, based on the event database 11a. The event specifying means 22 uses event identification information Ek (for example, an event name such as “home run”) in which the input cluster value data string is equal to the scene digitization string Sd of the event database 11a shown in FIG. The event of the continuous scene is identified, and the identification result (discrimination event) is output.

  Here, when the feature quantity extraction unit 14 outputs the video feature quantity of the scene to the feature quantity digitizing unit 21, the event specifying unit 22 is notified of the first frame number and the last frame number of the scene. I will do it. As a result, the event specifying unit 22 adds the first frame number of the first scene of the continuous scene and the last frame number of the last scene to the discrimination event, thereby obtaining the first frame number and the last frame number of the entire event. Can be output simultaneously.

  As described above, the video event determination device 2 can determine an event (video event) from an input video. Since this video event is discriminated by a numerical sequence in which video feature quantities are classified, the event can be automatically discriminated from the video scene itself, which could not be done conventionally.

  The video event determination device 2 can be realized by causing a general computer to execute a program and operating an arithmetic device or a storage device in the computer. This program (video event determination program) can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM for distribution.

[Operation of video event discriminator]
Next, referring to FIG. 6 (refer to FIG. 5 as appropriate), the operation of the video event determination device according to the present invention will be described. FIG. 6 is a flowchart showing the operation (event determination operation) of the video event determination device. Steps S31 to S36 are the same as steps S1 to S6 in the operation of the learning data generation device 1 (FIG. 2) described in FIG. 3, and thus the description thereof will be omitted, and the operations after step S37 will be described. .

  After step S36, the video event discriminating apparatus 2 uses the video feature quantity extracted by the feature quantity extraction means 14 as a result of the feature quantity digitizing means 21 referring to the feature quantity classification database 10a as learning data. The feature values are converted into classified cluster values (step S37).

  Then, in the video event determination device 2, the event identification unit 22 refers to the event database 11a, and identifies which event the data sequence of the cluster values sequentially converted in step S37 corresponds to. (Step S38), the event name, the first frame number, and the last frame number of the event are output as discrimination events (Step S39).

Then, the video event discriminating apparatus 2 determines whether or not the input video has ended (step S40), and if it has not ended (No in step S40), it returns to step S31 and continues the operation. If the video has ended (Yes in step S40), the operation (event determination operation) ends.
With the above operation, the video event determination device 2 can determine an event (video event) from the video based on the learning data (the feature amount classification database 10a and the event database 11a).

It is explanatory drawing for demonstrating the outline | summary of the video event discrimination | determination method based on this invention. It is a block diagram which shows the structure of the learning data generation apparatus for video event discrimination | determination based on this invention. It is a flowchart which shows the operation | movement which the learning data generation apparatus which concerns on this invention produces | generates the feature-value classification database. It is a flowchart which shows the operation | movement which the learning data generation apparatus which concerns on this invention produces | generates an event database. It is a block diagram which shows the structure of the structure of the video event discrimination | determination apparatus based on this invention. It is a flowchart which shows operation | movement of the video event discrimination | determination apparatus based on this invention. FIG. 2A is a diagram visualizing the tracking of a node in a node tracking unit, where FIG. 3A is a diagram illustrating a state in which a node is installed in a frame image, and FIG. . It is the conceptual diagram which showed the concept which classifies a node as the same class (cluster). It is a figure which shows an example of the data of the image | video feature-value data which a feature-value extraction means produces | generates. It is the conceptual diagram which showed the concept which simplifies the scene of an image | video by a some rectangular area. It is a structure figure which shows the example of the structure of a feature-value classification database. It is a structure figure which shows the example of the structure of a scene classification database. It is a structural diagram which shows the example of the structure of an event database.

Explanation of symbols

1 Learning data generator (learning data generator for video event discrimination)
2 Video event discriminating device 10 Feature quantity classification DB (database) storage means 10a Feature quantity classification database 11 Event DB (database) storage means 11a Event database 12 Scene division means 12a Event start detection means 13 Reference image storage means 13a Reference image 14 Features Quantity extraction means 15 Video feature quantity storage means 16 Feature quantity classification means 17 Scene digitization means 18 Scene classification DB (database) storage means 18a Scene classification database 19 Scene video reproduction means 20 Event setting means 21 Feature quantity digitization means 22 Event specification means

Claims (6)

A video event discriminating apparatus that discriminates the type of event that occurs in the video based on the video feature amount in each scene of the input video,
A feature quantity classification database storage unit storing a feature quantity classification database in which representative values of similar video feature quantities and numerical data for classifying the representative values are associated in advance;
An event database storage unit that stores an event database in which the event type and a scene digitization sequence in which a plurality of continuous scenes are represented by a data sequence of the numerical data are associated in advance;
Scene dividing means for dividing the input video for each scene;
Feature quantity extracting means for extracting the video feature quantity in the scene from a plurality of frame images included in the scene divided by the scene dividing means;
A feature quantity quantification means for referring to a feature quantity classification database stored in the feature quantity classification database storage means and converting the video feature quantity extracted by the feature quantity extraction means into the numerical data;
Referring to an event database stored in the event database storage means, an event specifying means for specifying an event type corresponding to a data string of numerical data converted by the feature value digitizing means;
A video event discriminating apparatus comprising: Reference image storage means for storing in advance a reference image that is the start of an event that occurs in the video,
Further, the scene division means includes event start detection means for detecting the start of an event in the video as a start point of the scene switching by comparing the frame image with the reference image. The video event discriminating apparatus according to claim 1. In order to determine the type of event that occurs in the video based on the video feature amount in each scene of the input video,
Scene dividing means for dividing the inputted video into scenes;
Feature quantity extracting means for extracting the video feature quantity in the scene from a plurality of frame images included in the scene divided by the scene dividing means;
With reference to a feature quantity classification database in which representative values of similar video feature quantities and numerical data for classifying the representative values are associated in advance, the video feature quantities extracted by the feature quantity extraction unit are Feature value digitizing means to convert to data,
Numeric data converted by the feature value digitizing means with reference to an event database in which the event type and a scene digitizing sequence in which a plurality of consecutive scenes are represented by a data sequence of the numeric data are associated in advance Event identification means for identifying the type of event corresponding to the data column of
A video event discriminating program characterized by functioning as A learning data generation device for video event determination that generates a feature amount classification database and an event database that are learning data used in the video event determination device according to claim 1,
Scene dividing means for dividing the input video for each scene;
Feature quantity extracting means for extracting the video feature quantity in the scene from a plurality of frame images included in the scene divided by the scene dividing means;
Feature quantity classification means for generating a feature quantity classification database that classifies video feature quantities extracted by the feature quantity extraction means in association with numerical data for each representative value of similar video feature quantities;
A scene digitizing means for associating, for each scene, a frame image number for identifying a frame image included in the scene and the numerical data of the video feature quantity classified by the feature quantity classifying means;
Scene video playback means for playing back the scene based on the frame image number for each scene associated with the scene digitizing means;
Event identification information indicating the type of event is input to a plurality of consecutive scenes reproduced by the scene image reproduction means, and the event identification information and the image feature amount corresponding to the plurality of scenes Event setting means for generating the event database in association with a scene digitization sequence that is a data sequence of
A learning data generating device for video event discrimination characterized by comprising: Reference image storage means for storing in advance a reference image that is the start of an event that occurs in the video,
Furthermore, the scene detection means includes an event start detection means for detecting the start of an event in the video as a start point of the scene change by comparing the frame image with the reference image. The learning data generation device for video event discrimination according to claim 4. In order to generate a feature amount classification database and an event database, which are learning data used in the video event determination device according to claim 1 or 2,
Scene dividing means for dividing the inputted video into scenes;
Feature quantity extracting means for extracting the video feature quantity in the scene from a plurality of frame images included in the scene divided by the scene dividing means;
Feature quantity classification means for generating a feature quantity classification database that classifies video feature quantities extracted by the feature quantity extraction means in association with numerical data for each representative value of similar video feature quantities;
A scene quantification unit that associates, for each scene, a frame image number for identifying a frame image included in the scene and the numerical data of the video feature amount classified by the feature amount classification unit;
Scene video playback means for playing back the scene based on the frame image number for each scene associated with the scene digitizing means;
Event identification information indicating the type of event is input to a plurality of consecutive scenes reproduced by the scene image reproduction means, and the event identification information and the image feature amount corresponding to the plurality of scenes Event setting means for generating an event database in association with a scene digitization sequence that is a data sequence of the numerical data of
A learning data generation program for discriminating video events, characterized in that

Images