JP4906588B2 - Specific operation determination device, reference data generation device, specific operation determination program, and reference data generation program - Google Patents

Description

  The present invention relates to video analysis technology, and more particularly to a specific motion determination device, a reference data generation device, a specific motion determination program, and a reference data generation program for determining a specific motion of a person included in a video.

  In general, in a pitching scene, a video (hereinafter referred to as a PC size video) shot from behind a pitcher is broadcast. Therefore, in order to extract a pitching scene from a baseball broadcast video, it seems possible to collect all pitching scenes by detecting a PC size video from the video. However, since the PC size video may be used other than the pitching scene such as a pitcher changing scene, it is not possible to extract only the pitching scene even if the PC size video is detected. The timing of the pitch cannot be determined.

  Therefore, conventionally, a method for searching for a desired scene from a video signal such as a television broadcast has been proposed. For example, a method is disclosed in which a baseball play start time and end time are determined by referring to pre-registered grasshopper face feature amounts and pitcher region transition information, and an on-play video is extracted. (See Patent Document 1).

In addition, a method has been proposed in which a scoring scene is detected by displaying a score by a character super near the center of the screen of a sports program, and a characteristic scene is detected by the amount of motion vectors of objects and backgrounds in a video (Patent Literature). 2).
JP 2006-238393 A (paragraph numbers 0034 to 0089) JP-A-11-55613 (paragraph numbers 0024 to 0035)

  However, the method described in Patent Document 1 is not suitable for real-time processing because the amount of information to be registered in advance is enormous and the processing load increases because facial feature amounts are registered. Further, since the vertical movement of the pitcher is a simple feature amount, there is a possibility that an operation other than the pitching may be erroneously determined as the pitching operation.

  In the method described in Patent Document 2, a specific scene is determined from a super display, but a specific scene cannot be determined from an image without a super display. Furthermore, since the determination using the motion vector is performed by simple threshold processing, there is a possibility that the determination may be erroneous.

  Most baseball events (for example, strikeout, hit, double play, etc.) start from the pitching scene. Therefore, in order to determine a baseball event, it is important to first identify a pitching scene. When such event information is added as metadata to a huge amount of past video, it is desired to automatically determine a scene of a specific action such as a throwing scene from this huge amount of video. It was.

  The present invention has been made to solve the above-described problems of the prior art, and is capable of determining a scene of a specific operation from a video with high accuracy without using a facial feature amount or a super display. An object is to provide an operation determination device, a reference data generation device, a specific operation determination program, and a reference data generation program.

  In order to solve the above-described problem, the specific motion determination device according to claim 1, wherein each time-series frame image in a video of a specific motion of a person is an adjacent frame that is another frame image continuous with the frame image. A shift of the entire frame image with respect to the image is corrected, and a difference image between the corrected frame image and the adjacent frame image is generated. Based on the difference image, the motion of the person region is corrected. Reference time-series data that stores in advance reference time-series data generated by calculating a plurality of feature quantities normalized according to a predetermined standard indicating the features, and arranging the feature quantities in time series for each video and performing dimension compression By referring to the reference time-series data from the storage device, an image of a specific action of a person included in the input video is input from the input video. A specific operation determination device determines, and the shooting direction correction means, a feature amount calculating means, the time-series data calculation means, configured to include an operation determination unit.

According to such a configuration, the specific operation determination device uses the shooting direction correction unit to perform, for each of the time-series frame images of the input video, the frame image for the adjacent frame image that is another frame image continuous with the frame image. By detecting and correcting the shift during the entire shooting, the feature amount calculating unit generates a difference image between the frame image corrected by the shooting direction correcting unit and the adjacent frame image, and the number of pixels in the difference region, The variance of the difference area, the moving speed of the center of gravity of the difference area, the number of difference pixels in the person area, and the inclination of the principal axis of the person area are normalized according to a predetermined reference and calculated as a plurality of feature amounts. In addition, the specific action determination device generates time-series data that is dimensionally compressed by time-series data calculation means by arranging the feature quantities calculated by the feature quantity calculation means in time series in the input video and performing principal component analysis. Then, the operation determination means calculates the correlation coefficient for all principal components of the reference time series data stored in the reference time series data storage device and the time series data generated by the time series data calculation means, In the time series data, a video corresponding to a section where the correlation coefficient is equal to or greater than a predetermined threshold is determined as a video of a specific operation.

  As a result, the specific action determination device can generate time-series data from the input video and determine the video of the specific action included in the input video. Here, reference time-series data is generated in advance using a video of a specific operation. Since this reference time series data is analyzed from the difference image between the frame images in the video of the specific action, it indicates the characteristics of the movement of the person's area in the video of the specific action. Is analyzed from the difference image between the frame images in the input video, and thus indicates the characteristics of the movement of the person region in the input video. Therefore, the specific operation determination device can determine, in the time series data, the video in the input video corresponding to the section having a high correlation with the reference time series data as the video of the specific operation.

  In addition, the reference data generation device according to claim 2 is used in the specific motion determination device according to claim 1, and reference time-series data indicating characteristics of the motion of the person area in the video of the specific motion of the person The reference data generation device for generating the image data includes a shooting direction correction unit, a feature amount calculation unit, and a time-series data calculation unit.

According to such a configuration, the reference data generation device uses the shooting direction correction unit to, for each of the time-series frame images in the video of the specific action of the person, an adjacent frame image that is another frame image continuous to the frame image. for, and detect and correct deviation of the time the frame image overall shooting, by the feature amount calculating means, and the frame image corrected by the photographing direction correction means generates a difference image between adjacent frame images, the difference The number of pixels in the area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the number of difference pixels in the person area, and the inclination of the principal axis of the person area are normalized according to a predetermined criterion, and a plurality of features It is calculated as the amount. The reference data generating device, when the series data calculation means, a feature amount calculated by the feature calculating unit by performing side-by-side principal component analysis on the time series in the image, generating a reference time-series data dimension reduction To do.

  Accordingly, the reference data generation device can generate reference time-series data indicating the characteristics of the movement of the person's area in the video of the specific operation from the video of the specific operation.

  Furthermore, the specific motion determination program according to claim 3 is configured to perform, for each time-series frame image in a video of a specific motion of a person, the frame for an adjacent frame image that is another frame image continuous with the frame image. A predetermined image indicating a characteristic of the movement of the person's area based on the difference image by generating a difference image between the corrected frame image and the adjacent frame image by correcting a shift during photographing of the entire image. From a reference time-series data storage device that pre-stores reference time-series data generated by calculating a plurality of feature quantities normalized according to a reference and dimensionally compressing the feature quantities in time series for each video. Referring to the reference time-series data, a video of a specific action of a person included in the input video is determined from the input video. In order, a computer, photographing direction correction means, feature amount calculating means, the time-series data calculation means, and a function as an operation determination unit.

According to such a configuration, the specific operation determination program causes the shooting direction correction unit to perform, for each of the time-series frame images of the input video, the frame image for the adjacent frame image that is another frame image continuous to the frame image. By detecting and correcting the shift during the entire shooting, the feature amount calculating unit generates a difference image between the frame image corrected by the shooting direction correcting unit and the adjacent frame image, and the number of pixels in the difference region, The variance of the difference area, the moving speed of the center of gravity of the difference area, the number of difference pixels in the person area, and the inclination of the principal axis of the person area are normalized according to a predetermined reference and calculated as a plurality of feature amounts. In addition, the specific action determination program generates a dimensionally compressed time series data by performing a principal component analysis by arranging the feature quantities calculated by the feature quantity calculation means in time series in the input video by the time series data calculation means. Then, the operation determination means calculates the correlation coefficient for all principal components of the reference time series data stored in the reference time series data storage device and the time series data generated by the time series data calculation means, In the time series data, a video corresponding to a section where the correlation coefficient is equal to or greater than a predetermined threshold is determined as a video of a specific operation.

  As a result, the specific action determination program can generate time-series data from the input video and determine the video of the specific action included in the input video. Here, by using the video of the specific motion and generating the reference time-series data indicating the characteristics of the movement of the person's area in the video of the specific motion in advance, the specific motion determination program The video corresponding to the section having a high correlation with the reference time-series data can be determined as the video of the specific operation.

  According to a fourth aspect of the present invention, there is provided a reference data generation program that is used in the specific action determination device according to the first aspect, and that is a reference time-series data indicating the characteristics of the movement of the person area in the video of the specific action of the person. Therefore, the computer is caused to function as an imaging direction correction unit, a feature amount calculation unit, and a time series data calculation unit.

According to such a configuration, the reference data generation program uses the shooting direction correction unit to, for each of the time-series frame images in the video of the specific action of the person, an adjacent frame image that is another frame image continuous to the frame image. for, and detect and correct deviation of the time the frame image overall shooting, by the feature amount calculating means, and the frame image corrected by the photographing direction correction means generates a difference image between adjacent frame images, the difference The number of pixels in the area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the number of difference pixels in the person area, and the inclination of the principal axis of the person area are normalized according to a predetermined criterion, and a plurality of features It is calculated as the amount. The reference data generation program, when the series data calculation means, a feature amount calculated by the feature calculating unit by performing side-by-side principal component analysis on the time series in the image, generating a reference time-series data dimension reduction To do.

  Thus, the reference data generation program can generate reference time-series data indicating the characteristics of the movement of the person's area in the video of the specific operation from the video of the specific operation.

  The specific operation determination device, the reference data generation device, the specific operation determination program, and the reference data generation program according to the present invention have the following excellent effects.

  According to the first or third aspect of the invention, it is possible to determine a video of a specific operation included in the input video based on the time-series data of the input video. Then, unlike the prior art, it is not necessary to identify a face, and the feature amount is dimensionally compressed when generating time-series data, so that the processing load can be reduced. Further, since there is no need to use additional information added to the video later as in the conventional super display, the present invention can also be applied to an input video to which no additional information is added. Also, based on time-series data indicating the motion characteristics of the person's area in the input video, it is determined whether the video is a specific motion of the person with high accuracy in order to determine whether the video is a specific motion of the person. can do.

  According to the second or fourth aspect of the present invention, reference time-series data can be generated from a video of a specific operation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, a specific action extraction system including a reference data generation apparatus and a specific action determination apparatus according to the present invention will be described.

[Configuration of specific action extraction system]
First, the configuration of the specific action extraction system S will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of a specific action extraction system including a reference data generation apparatus and a specific action determination apparatus according to the present invention.

  The specific action extraction system S generates reference time-series data indicating characteristics of a person's movement due to the action based on the specific action video that is a plurality of past videos in which the specific action of the person is captured, Based on the reference time-series data, the video of the operation included in the input video is determined from the input video input from the outside. The specific action extraction system S includes a reference data generation device 1, a reference time-series data storage device 3, and a specific action determination device 5. Here, a case where the specific motion extraction system S extracts a baseball pitching motion image (scene) will be described as an example, but a video shot in a state close to fixed shooting in which the camera is fixed. If so, the present invention can be applied not only to a pitching operation but also to other operations of baseball, various sports operations, and operations other than sports.

  The reference data generation device 1 generates reference time-series data indicating the characteristics of a person's movement caused by the operation based on a specific operation image input from the outside. The reference time series data generated here is stored in the reference time series data storage device 3.

  The specific motion video input here is a video of a specific motion of a person. For example, only a section from the start to the end of a specific motion is manually extracted from a video shot in the past. It is a picture. The reference time-series data indicates the characteristics of the person's movement in the specific motion video, and is data serving as a reference for determining whether the video is a specific motion in the specific motion determination device 5 described later. is there.

  The reference time series data storage device 3 stores the reference time series data generated by the reference data generation device 1, and is a general storage means such as a hard disk. This reference time series data is referred to and used by the specific operation determination device 5.

  The specific motion determination device 5 determines a video of a specific motion of a person from an input video input from the outside based on the reference time-series data stored in the reference time-series data storage device 3. Here, information indicating the section of the video determined as the video of the specific operation is output to the outside.

  Here, the specific action determination device 5 generates time-series data indicating the characteristics of the person's movement in the input video by the same method as the reference data generation device 1 generates the reference time-series data. Then, the reference time series data stored in the reference time series data storage device 3 is used as a learning result obtained by learning the feature amount of the video of the specific operation, and the time series data having a high correlation with the reference time series data is obtained. The video of the corresponding section is determined as the video of a specific operation. It is assumed that the reference time-series data is already stored in the reference time-series data storage device 3 when the specific operation determination device 5 determines the input video.

Hereinafter, detailed configurations of the reference data generation device 1 and the specific operation determination device 5 will be described.
(Configuration of reference data generator)
A configuration of the reference data generation device 1 will be described. The reference data generation apparatus 1 includes an image storage unit 11, a shooting direction correction unit 12, a feature amount calculation unit 13, and a time series data calculation unit 14.

  The image storage means 11 is a memory for storing a specific operation video input from the outside and performing various signals / image processing, for example, recording the specific operation video as digital data in units of one frame. is there. The frame image stored here is delayed by one frame and output to the shooting direction correction unit 12 and the feature amount calculation unit 13.

The photographing direction correction unit 12 is a frame image (adjacent frame image Image) input from the image storage unit 11 of a frame image of an input signal input from the outside (hereinafter referred to as a current frame image Image _t ). _This is to correct a shift of the image due to a shift of the photographing direction with respect to _t-1 ). The current frame image Image _t corrected here is output to the feature amount calculating means 13.

Here, in many cases, the shooting direction of a camera shot during a baseball broadcast does not change so that it can be regarded as fixed shooting, but in reality the camera is shaken by the cameraman's hand shake, the camera installation position vibration, etc. There is a slight shaking. Then, the feature value calculation means 13 described later generates a difference image between the current frame image Image _t and the adjacent frame image Image _t−1, and detects a motion in the person area in the current frame image Image _t . At this time, if the current frame image Image _t is deviated from the adjacent frame image Image _{t-1 due} to camera shake, it is difficult to accurately analyze the movement of the person area. Therefore, here, the photographing direction correction means 12, it was decided to calculate the motion vectors for the adjacent frame images Image _t-1 of the current frame image Image _t, to correct the position of the entire current frame image Image _t.

This motion vector can be calculated by, for example, a block matching method. Here, with reference to FIG. 2 (refer to FIG. 1 as appropriate), a method in which the shooting direction correction unit 12 calculates a motion vector by the block matching method and corrects the shift of the current frame image Image _t will be described. FIG. 2 is a schematic diagram schematically illustrating an example of a current frame image and a target rectangular area in the current frame image, FIG. 2A is a schematic diagram schematically illustrating an example of the current frame image, and FIG. FIG. 6B is a schematic diagram schematically illustrating an example of a target rectangular area in a current frame image indicated by B in FIG.

First, the shooting direction correction unit 12 sets a target rectangular area B in the current frame image Image _t shown in FIG. Here, the photographing direction correcting unit 12 sets the target rectangular area B based on a command from the operator. Here, in the baseball broadcast pitch scene, the shooting direction is almost fixed, so the background of the signboard and other features always appears at the same position. Therefore, in one baseball broadcast, the operator can use the same area without changing it once the position and size of the area (the target rectangular area B) are designated once. At this time, it is preferable to designate a region that does not cover the player's region as much as possible. Note that the shooting direction correction unit 12 may automatically set a rectangular region of interest B having a predetermined size at a predetermined position, for example, without a command from the operator.

Assume that a rectangular region of interest B as shown in FIG. 2B is set. Then, the shooting direction correction unit 12 obtains a location that matches the image of the target rectangular region B from the adjacent frame image Image _t-1 delayed by one frame input from the image storage unit 11 by block matching. A motion vector of the target rectangular area B is obtained. Here, when the motion vector is, for example, (1, 2), the shooting direction correcting unit 12 moves all the pixels of the current frame image Image _t by (−1, −2), thereby moving the current frame image. The shift between Image _t and adjacent frame image Image _t-1 can be corrected. Here, the current direction image Image _t and the adjacent frame image Image _t-1 constituting the specific operation video are sequentially input to the shooting direction correction unit 12, and the shooting direction correction unit 12 performs the above processing for each frame. Do.

Returning to FIG. 1, the description will be continued. Based on the current frame image Image _t input from the shooting direction correction unit 12 and the adjacent frame image Image _t−1 input from the image storage unit 11, the feature amount calculation unit 13 moves the person in the specific motion video. The feature amount indicating the feature is calculated. The feature amount calculated here is output to the time-series data calculation unit 14. Here, the feature amount calculation means 13 generates a difference image between the current frame image Image _t and the adjacent frame image Image _t−1, and calculates the feature amount in this image.

Here, the feature amount calculation means 13 generates a binarized image indicating the difference between the current frame image Image _t and the adjacent frame image Image _t−1 and calculates the feature amount. Hereinafter, a method in which the feature amount calculation unit 13 generates a binarized image indicating the difference and calculates the feature amount will be described.

Here, the pixel value at the time of calculating the difference by the feature amount calculating unit 13 may be luminance or a specific color such as red, blue, or green. Then, the feature amount calculating unit 13 generates a binarized image in which the pixel value of the region where the absolute value of the calculated difference is equal to or greater than the threshold is “1” and the pixel value of the other region is “0”. . As a result, it is possible to extract only an area (for example, an arm at the time of pitching) in which movement has occurred between the current frame image Image _t and the adjacent frame image Image _t-1 .

  Further, the feature amount calculation means 13 calculates a feature amount in the image from the obtained difference image (binarized image). Here, in the baseball video, the shooting direction changes slightly for each pitch, so a difference occurs, and the image coordinates of the region extracted as the region where the motion has occurred (hereinafter referred to as the difference region) is directly used as the feature amount. I can't. For example, the image coordinates of the center of gravity of the home base change as the shooting direction changes. Therefore, the feature quantity calculation means 13 calculates the following feature quantity that excludes the influence of the change in the shooting direction in consideration of the change in the shooting direction. Note that these feature amounts are merely examples, and the feature amount calculation unit 13 may calculate other feature amounts indicating the characteristics of the person's movement.

  Here, the feature amount calculation means 13 calculates, from the difference image, the number of pixels in the difference area (hereinafter referred to as the difference pixel number), the variance of the difference area, the moving speed of the centroid of the difference area (hereinafter referred to as the centroid speed), The number of difference pixels in the pitcher area and the inclination of the main axis of the pitcher area are calculated as feature amounts. Here, with reference to FIG. 3, a method by which the feature amount calculation unit 13 calculates each feature amount will be described. FIG. 3 is a schematic diagram schematically showing the difference image generated by the feature amount calculation means. In FIG. 3, the difference area is shown in white.

<Number of difference pixels>
The difference pixel number is the total number of pixels in the difference area in the entire difference image. Here, the feature amount calculating unit 13 obtains the total number of pixels whose pixel value of the difference image is “1”.

<Distribution of difference area>
The variance of the difference area is the variance of the difference from the center of gravity C of the difference area. Here, the feature amount calculating means 13 calculates the horizontal dispersion and the vertical dispersion.

<Center of gravity speed>
The center-of-gravity speed is the amount of movement of the center of gravity C of the difference area from the center of gravity C of the difference area of the previous frame. Here, the feature amount calculation unit 13 calculates the horizontal movement amount and the vertical movement amount of the center of gravity C.

<Number of difference pixels in pitcher area>
For example, as shown in FIG. 3, the pitcher area Q is an area within a certain number of pixels (for example, 2σ with σ being a standard deviation) in the horizontal and vertical directions from the center of gravity C of the difference area. This is the number of pixels in the difference area in the pitcher area Q at the time. Further, here, the feature amount calculation means 13 equally divides the pitcher area Q into four, and obtains the number of difference pixels for each of the upper left area Q1, the upper right area Q2, the lower left area Q3, and the lower right area Q4. .

<Inclination of the main axis of the pitcher area>
The inclination of the main axis of the pitcher area is the inclination of the main axis M of the difference area in the pitcher area R. Here, the feature quantity calculating means 13 can calculate the inclination of the principal axis of the pitcher area by obtaining the second moment of the image in the pitcher area R. Here, the p + q moment _Mpq of the image I (x, y) is expressed by the following equation (1). And the feature-value calculation means 13 can calculate inclination (theta) of the main axis | shaft of a pitcher area | region by the following formula | equation (2).

Furthermore, the feature amount calculation unit 13 normalizes each value calculated as described above within a range of 0 to 1 to obtain a feature amount of the difference image. For example, the feature amount calculating unit 13 can calculate the normalized difference pixel number s_cnt by the following equation (3). Here, NI is the total number of pixels of the difference image, and NS is the number of difference pixels.
s_cnt = NS / NI (3)

  Returning to FIG. 1, the description will be continued. The time series data calculation means 14 arranges the feature quantities for each specific motion video calculated by the feature quantity calculation means 13 in time series and performs dimension compression. Here, the reference time series data generated by the dimension compression is stored in the reference time series data storage device 3.

  Here, the pitching operation is often completed in about 3 seconds. For example, in the case of a specific action video for 3 seconds, the specific action video is 90 frames (3 [seconds] × 30 [frames / second]). The feature amounts are arranged in time series. However, if all the feature amounts calculated by the feature amount calculation means 13 are used, the number of dimensions is high (here, 11 dimensions) and the processing load is large. Therefore, the time-series data calculation unit 14 performs principal component analysis on the feature amount and compresses the dimension. The number of dimensions after compression is preferably about 2 to 5 dimensions. Hereinafter, dimension compression by the time-series data calculating unit 14 will be described.

The time-series data calculation unit 14 generates a feature quantity matrix A as shown in the following equation (4). Here, a _tj (t = 0 to T, j = 1 to N) is the j-th feature amount at time t and is a component of the matrix A. T represents the time length of the video, and N represents the number of types of feature values (number of dimensions).

The time-series data calculating unit 14 performs principal component analysis on the matrix A to obtain eigenvalues P _{0 to} P _T and eigen vectors V _{0 to} V _T shown in the following equation (5). In Expression (5), the eigenvalues P _{0 to} P _T are arranged from the top in descending order. _{Here, v ij (i = 0~T,} j = 1~N) is a component of the eigenvector _{V t.} The time-series data calculating unit 14 can compress the number of dimensions to K by using only the principal components having eigenvalues P _{0 to} P _K (K <T) that are equal to or greater than a certain value.

  Further, the time series data calculation means 14 calculates the principal component score by calculating the inner product of the eigenvector and the feature amount for each frame, and the time series data arranged in time series is referred to as the reference time series data. To do. An example of reference time series data is shown in FIG. Here, the number of dimensions after the dimension compression is 3, and FIG. 4 shows an example of reference time series data composed of time series data s1 to s3 of three-dimensional principal component scores. FIG. 4 is a graph showing an example of reference time series data calculated by the time series data calculation means. In FIG. 4, the horizontal axis represents the number of frames (time), and the vertical axis represents the main component score.

  As described above, the reference data generation device 1 generates reference time-series data as a reference for determining whether or not the video is a specific motion in the specific motion determination device 5 to be described later, from the specific motion video input from the outside. can do.

(Configuration of specific operation determination device)
Next, the configuration of the specific action determination device 5 will be described. The specific motion determination device 5 includes an image storage unit 51, a shooting direction correction unit 52, a feature amount calculation unit 53, a time series data calculation unit 54, an input time series data storage unit 55, and a motion determination unit 56. Prepare.

  The image storage means 51 is a memory for storing input video input from the outside and performing various signal / image processing. For example, the input video is recorded as digital data in units of one frame. The frame image stored here is delayed by one frame and output to the shooting direction correction unit 52 and the feature amount calculation unit 53.

The photographing direction correcting means 52 is a frame image (adjacent frame image Image) input from the image storage means 51 of a frame image of the input signal input from the outside (hereinafter referred to as current frame image Image _t ). _This is to correct a shift of the image due to a shift of the photographing direction with respect to _t-1 ). The current frame image Image _t corrected here is output to the feature amount calculating means 53. Note that this shooting direction correction means 52 is compared with the shooting direction correction means 12 of the reference data generation device 1 in that the input video includes a video of a specific operation, such as a baseball broadcast video, for example. Thus, the function is the same only when the output destination is the feature amount calculation means 53.

Based on the current frame image Image _t input from the shooting direction correction unit 52 and the adjacent frame image Image _t−1 input from the image storage unit 51, the feature amount calculation unit 53 determines the movement of the person in the input video. A feature amount indicating a feature is calculated. The feature amount calculated here is output to the time-series data calculation unit 54. Note that the feature amount calculation means 53 is compared with the feature amount calculation means 13 of the reference data generation device 1 in that the input video includes a video of a specific operation such as a baseball broadcast video, for example. Thus, the type and the calculation method of the feature amount to be calculated are the same only when the output destination is the time-series data calculation unit 54.

  The time series data calculation unit 54 arranges the feature amounts for each input video calculated by the feature amount calculation unit 53 in time series and performs dimension compression. Here, the time series data generated by dimension compression is stored in the input time series data storage means 55. Note that the time series data calculation means 54 is characterized in that the input feature quantity is the feature quantity of the input video and the generated time series data is the input time series as compared to the time series data calculation means 14 of the reference data generation apparatus 1. The dimensional compression method and the number of dimensions after compression are the same only by storing in the data storage means 55.

  Here, when the specific motion determination device 5 determines the video of the specific motion of the person from the input video in real time, the time-series data calculation unit 54 receives the feature amount for one frame from the feature amount calculation unit 53. Each time an input is made, time series data may be generated by dimensionally compressing feature quantities that are traced back by a predetermined number (for example, the number of frames corresponding to several seconds) from this frame in time series.

  The input time series data storage means 55 stores the time series data generated by the time series data calculation means 54 and is a general storage means such as a hard disk. This time series data is referred to and used by the operation determination means 56.

  The operation determination means 56 calculates a correlation coefficient between the reference time series data stored in the reference time series data storage device 3 and the time series data stored in the input time series data storage means 55, and is included in the input video. The video of a specific action (throwing) is determined and detected. Here, information indicating the section of the video determined as the video of the specific operation is output to the outside.

Here, the motion determination means 56 can calculate the correlation coefficient r _x for a certain principal component _x by the following equation (6). Here, I _t is, principal component score of the principal component x of the time series data at time t, R _t is the principal component score of the principal component x of the reference time-series data at time t.

Then, the operation determination unit 56 calculates a correlation coefficient for all the principal components, and adds the weights according to the eigenvalues to add a final correlation coefficient. Here, the motion determination means 56 can calculate the final correlation coefficient R by the following equation (7). Note that w _x is a weight of the principal component x, and P _x is an eigenvalue.

The operation determination unit 56 calculates the correlation coefficient R for the entire time series data section by shifting the correlation coefficient R by one frame from the time series data. When the specific motion determination device 5 determines the video of the specific motion of the person from the input video in real time, the time corresponding to a predetermined number of frames in which the start time is shifted frame by frame by the time-series data calculation unit 54. Since the series data is generated, the operation determination unit 56 may calculate the correlation coefficient r _x with respect to each time series data by the equation (6). Then, the motion determination means 56 determines a part of the input video corresponding to the section where the correlation coefficient exceeds a predetermined threshold in the time series data as the video of the specific motion, and the section of the video is Outputs information about the indicated time (frame number).

  As a result, the specific operation determination device 5 can determine the video of the specific operation from the input video based on the reference time-series data stored in the reference time-series data storage device 3.

  The configuration of the specific action extraction system S has been described above, but the present invention is not limited to this. For example, here, a case where a baseball pitching scene is determined has been described as an example. However, the reference data generation device 1 and the specific motion determination device 5 of the present invention are applicable not only to baseball but also to various motions. It is. For example, the specific action extraction system S can determine whether or not a corner kick has been performed by analyzing a camera image obtained by photographing the vicinity of the corner of the sucker. Also, for example, in the field of security, it is possible to determine abnormal behavior by applying it to video from a security camera or the like.

  Furthermore, by adding metadata to the video determined by the specific action extraction system S, it is possible to automatically generate metadata. It is also possible to promote the understanding of the situation to the viewer by presenting the obtained information on a baseball game. In particular, it can contribute to supporting video production for television broadcasting, data broadcasting, and the Internet.

  Note that the reference data generation device 1 may generate reference time-series data for each person by sequentially inputting a specific action video associated with a person (pitcher) performing a specific action from the outside. Good. At this time, the reference time-series data storage device 3 stores the reference time-series data for each person in association with identification information (for example, a person name or pitcher name) for identifying the person. Further, the action determination unit 56 of the specific action determination device 5 analyzes the correlation between the generated time series data and the reference time series data corresponding to each person. Then, the motion determination unit 56 can determine a video corresponding to time-series data having a high correlation with reference time-series data associated with identification information of a certain person as a video of a specific motion of the person. As described above, the specific motion determination device 5 can identify a person and determine a video of a specific motion, and outputs information on the time of the video of the pitch determined to be a video of the specific motion together with the identification information of the person. be able to.

  Further, when the reference data generation device 1 generates the reference time-series data, for example, the reference time-series data is associated with the time information of the timing of pitching of the specific motion video specified by the operator in the reference time-series data. It may be stored in the series data storage device 3. At this time, the motion determination means 56 of the specific motion determination device 5 corresponds to the time of the pitching timing associated with the reference time-series data in the video corresponding to the time-series data having a high correlation with certain reference time-series data. By setting the time as the pitching timing in the video, the specific motion determination device 5 can also determine and output the pitching timing of the video determined as the video of the specific motion.

  Further, when the reference data generating device 1 generates the reference time-series data, in addition to the time information of the pitching timing, the position of the person area (corresponding to the time calculated by the feature amount calculating unit 13 ( The center of gravity) information may also be stored in the reference time series data storage device 3 in association with the reference time series data. At this time, the motion determination means 56 of the specific motion determination device 5 uses the video of the pitch timing associated with the reference time-series data in the video corresponding to the time-series data highly correlated with certain reference time-series data. And the position of the person (pitcher) in the frame image at that time can also be output.

  Then, the same input video is input as the time information indicating the section of the video determined as the video of the specific motion by the specific motion determination device 5, the time information of the pitching timing, and the position information of the person. Then, it may be output to a pitching trajectory drawing device (see, for example, JP-A-2005-123824, not shown) that extracts and tracks the ball from the video of the pitching and generates a trajectory image indicating the pitching trajectory. This pitching trajectory drawing device sets a narrow search range for searching for a ball in each frame image of the input video, and can extract and track the ball from the search range in real time. Even if only a baseball broadcast video is directly input to such a pitch trajectory drawing device, this device cannot detect the presence or timing of the pitch and the timing of the pitch, and the broadcast camera performs the camera operation between pitches. It was necessary to manually set the pitching timing and the initial position of the ball search range each time.

  Therefore, the specific motion determination device 5 inputs time information indicating a video segment determined to be a video of a specific motion, pitch timing time information, and person position information to the pitch trajectory drawing device. Then, the pitching trajectory drawing device draws a trajectory image for the segment of the input video based on the time information indicating the segment of the video input from the specific motion determination device 5, and indicates the timing of the pitching , And the position of the person is regarded as an approximate release point of the ball, and the search range is set based on the position information, whereby the specific motion determination device 5 and the pitching trajectory drawing device It is possible to automatically perform the generation of the trajectory image by using. Furthermore, since broadcast camera images can be used as input images to be input to the specific motion determination device 5 and the pitching trajectory drawing device, there is no need to install a new camera or calibration, and operability is high.

  However, in the case of generating a trajectory image in real time, the specific motion determination device 5 of the present invention determines the motion after the pitching is completed, and therefore the determination result is input to the pitching trajectory drawing device. Will pass the timing at which the pitch trajectory drawing device initially sets the search range. Therefore, by inputting a video delayed by several seconds compared to the input video input to the specific motion determination device 5 to the pitch trajectory drawing device, after the determination result from the specific motion determination device 5 is input, the pitch trajectory drawing device However, the search range can be initialized in the frame image of the delayed input video, and the ball can be extracted and tracked.

  Note that the reference data generation device 1 and the specific operation determination device 5 can also realize each unit as a function program in a computer, and combine the function programs as a reference data generation program and a specific operation determination program. It is also possible to operate.

[Operation of specific action extraction system]
Next, referring to FIGS. 5 and 6 (refer to FIG. 1 as appropriate), the operation of the specific action extraction system S including the reference data generation device 1 and the specific action determination device 5 in the present invention will be described. FIG. 5 is a flowchart showing the reference data generation operation of the reference data generation apparatus according to the present invention. FIG. 6 is a flowchart showing the specific operation determination operation of the specific operation determination device according to the present invention.

(Reference data generation operation)
First, with reference to FIG. 5 (refer to FIG. 1 as appropriate), a reference data generation operation in which the reference data generation device 1 generates reference data serving as a determination criterion for a specific operation in the determination by the specific operation determination device 5 will be described. .

The reference data generating apparatus 1 receives a specific operation video from outside by the shooting direction correction unit 12 and adjoins the current frame image Image _t of the specific operation video delayed by one frame input from the image storage unit 11. The shift of the image due to the shift of the shooting direction with respect to the frame image Image _t-1 is corrected (step S11). Subsequently, the reference data generation device 1 uses the feature amount calculation unit 13 to obtain a difference image between the current frame image Image _t corrected in step S11 and the adjacent frame image Image _t−1 input from the image storage unit 11. The feature amount indicating the feature of the motion of the person in the specific motion video is calculated from the difference image (step S12).

  Then, the reference data generation device 1 determines whether the calculation of the feature amount is completed for all the frame images of the specific motion video by the shooting direction correction unit 12 (step S13). If not completed (No in step S13), the reference data generation device 1 returns to step S11, and the shooting direction correction unit 12 performs operations after the operation of correcting the shift of the next frame image. Do.

  On the other hand, when the processing is completed (Yes in step S13), the reference data generation device 1 uses the time-series data calculation unit 14 to arrange the feature amounts of the specific motion video calculated in step S12 in time series and to perform dimension compression. Then, reference time-series data is generated (step S14). Then, the reference data generation device 1 stores the reference time series data generated in step S14 in the reference time series data storage device 3 by the time series data calculation means 14 (step S15), and ends the operation.

(Specific operation judgment operation)
Further, referring to FIG. 6, the specific operation determination device 5 uses the reference time series data generated by the reference data generation operation shown in FIG. 5 to determine the specific operation video from the input video. The operation will be described.

The specific operation determination device 5 receives an input video from outside by the shooting direction correction unit 52, and the adjacent frame image delayed by one frame input from the image storage unit 51 of the current frame image Image _t of the input video. The shift of the image due to the shift of the shooting direction with respect to Image _t-1 is corrected (step S51). Subsequently, the specific action determination device 5 obtains a difference image between the current frame image Image _t corrected in step S51 and the adjacent frame image Image _t−1 input from the image storage unit 51 by the feature amount calculation unit 53. A feature amount indicating the feature of the motion of the person in the input video is calculated from the difference image (step S52).

  Then, the specific operation determination device 5 determines whether or not all the frame images of the input video have been completed by the shooting direction correction unit 52 (step S53). If not completed (No in step S53), the specific operation determination device 5 returns to step S51, and the shooting direction correction unit 52 performs the operation after the operation of correcting the shift of the next frame image. Do.

  On the other hand, when the processing is completed (Yes in step S53), the specific action determination device 5 arranges the feature quantities of the input video calculated in step S52 in time series and performs dimension compression by the time series data calculation unit 54, Time series data is generated (step S54). The specific action determination device 5 stores the time series data generated in step S54 by the time series data calculation unit 54 in the input time series data storage unit 55 (step S55). Furthermore, the specific action determination device 5 uses the action determination means 56 to correlate the time series data stored in step S55 and each reference time series data generated by the reference data generation operation (see FIG. 5). Is calculated, and a video whose correlation coefficient is equal to or greater than a threshold is determined as a video of a specific motion (step S56), and the motion is terminated.

It is the block diagram which showed the structure of the specific action extraction system provided with the reference data generation apparatus and specific action determination apparatus of this invention. Schematic diagram schematically showing an example of the current frame image and a rectangular region of interest in the current frame image, (a) is a schematic diagram schematically showing an example of the current frame image, and (b) is (a). It is a schematic diagram which shows typically the example of the attention rectangular area in the present frame image shown by B of FIG. It is a schematic diagram which shows typically the difference image produced | generated by the feature-value calculation means of the reference data generation apparatus of this invention. It is a graph which shows the example of the reference time series data calculated by the time series data calculation means of the reference data generation apparatus of this invention. It is the flowchart which showed the reference data production | generation operation | movement of the reference data production | generation apparatus in this invention. It is the flowchart which showed the specific operation | movement determination operation | movement of the specific operation determination apparatus in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS S Specific motion extraction system 1 Reference data production | generation apparatus 11 Image memory | storage means 12 Shooting direction correction means 13 Feature-value calculation means 14 Time series data calculation means 3 Reference time series data storage device 5 Specific motion determination apparatus 51 Image storage means 52 Shooting direction correction Means 53 Feature amount calculation means 54 Time series data calculation means 55 Input time series data storage means 56 Operation determination means

Claims (4)

For each time-series frame image in a video of a specific action of a person, the deviation at the time of shooting the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image is corrected and corrected. Generating a difference image between the frame image and the adjacent frame image, and calculating a plurality of feature amounts normalized based on a predetermined reference indicating a feature of the movement of the person based on the difference image; An input video input by referring to the reference time-series data from a reference time-series data storage device that pre-stores reference time-series data generated by dimensionally compressing the feature quantities in time series for each video. A specific action determination device for determining a video of a specific action of a person included in the input video,
Shooting direction correction means for detecting and correcting a shift in shooting of the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image for each of the time-series frame images of the input video; ,
A difference image between the frame image corrected by the photographing direction correction unit and the adjacent frame image is generated, and the number of pixels in the difference area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the person speed difference pixels in the area and the inclination of the main axis of the person area is normalized by the predetermined reference, the feature amount calculating means for calculating a plurality of feature amounts,
Time-series data calculating means for generating the dimension-compressed time-series data by arranging the feature quantities calculated by the feature-value calculating means in time series in the input video and performing principal component analysis ;
Calculating a correlation coefficient for all principal components of the reference time-series data stored in the reference time-series data storage device and the time-series data generated by the time-series data calculating means; In the above, the operation determination means for determining the video corresponding to the section in which the correlation coefficient is equal to or greater than a predetermined threshold as the video of the specific operation,
A specific action determination device comprising: A reference data generation device that generates reference time-series data indicating characteristics of movement of the person's area in a video of a specific action of a person used in the specific action determination device according to claim 1,
For each of the time-series frame images in the video of the specific action of the person, a shift at the time of shooting the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image is detected and corrected. Photographing direction correction means;
A difference image between the frame image corrected by the photographing direction correction unit and the adjacent frame image is generated, and the number of pixels in the difference area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the person A feature amount calculating means for normalizing the difference pixel number of the region and the inclination of the principal axis of the person region according to a predetermined reference, and calculating as a plurality of feature amounts;
Time series data calculation means for generating the reference time series data subjected to dimension compression by arranging the feature quantities calculated by the feature quantity calculation means in time series in the video and performing principal component analysis ;
A reference data generation device comprising: For each time-series frame image in a video of a specific action of a person, the deviation at the time of shooting the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image is corrected and corrected. Generating a difference image between the frame image and the adjacent frame image, and calculating a plurality of feature amounts normalized based on a predetermined reference indicating a feature of the movement of the person based on the difference image; An input video input by referring to the reference time-series data from a reference time-series data storage device that pre-stores reference time-series data generated by dimensionally compressing the feature quantities in time series for each video. In order to determine a video of a specific action of a person included in the input video,
For each of the time-series frame images of the input video, a shooting direction correction unit that detects and corrects a shift during shooting of the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image;
A difference image between the frame image corrected by the photographing direction correction unit and the adjacent frame image is generated, and the number of pixels in the difference area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the person speed difference pixels in the area and the inclination of the main axis of the person area is normalized by predetermined reference, the feature amount calculating means for calculating a plurality of feature quantities,
Time-series data calculating means for generating dimensionally compressed time-series data by arranging the feature quantities calculated by the feature-quantity calculating means in time series in the input video and performing principal component analysis ;
Calculating a correlation coefficient for all principal components of the reference time-series data stored in the reference time-series data storage device and the time-series data generated by the time-series data calculating means; In, the operation determination means for determining the video corresponding to the section where the correlation coefficient is equal to or greater than a predetermined threshold as the video of the specific operation,
A specific operation determination program characterized in that it functions as a program. A computer for generating reference time-series data indicating characteristics of movement of the person area in a video of a specific action of a person used in the specific action determination device according to claim 1,
For each of the time-series frame images in the video of the specific action of the person, a shift at the time of shooting the entire frame image with respect to an adjacent frame image that is another frame image continuous with the frame image is detected and corrected. Shooting direction correction means,
A difference image between the frame image corrected by the photographing direction correction unit and the adjacent frame image is generated, and the number of pixels in the difference area, the variance of the difference area, the moving speed of the center of gravity of the difference area, the person A feature amount calculating means for normalizing the difference pixel number of the region and the inclination of the principal axis of the person region according to a predetermined reference, and calculating as a plurality of feature amounts;
Time series data calculation means for generating the reference time series data subjected to dimension compression by arranging the feature quantities calculated by the feature quantity calculation means in time series in the video and performing principal component analysis ;
A reference data generation program characterized by functioning as

Images