JP6261222B2 - Identification device and identification program - Google Patents

Description

  The present invention relates to an apparatus and a program for identifying an operation indicated by an image.

  2. Description of the Related Art Conventionally, there is a technique for recognizing an object from a still image by combining a SIFT (Scale-Invariant Feature Transform) algorithm that detects a feature point from an image and describes a feature amount, and a BoF (Bag-of-Features) method. BoF is a method of integrating extracted local feature values and expressing them as a histogram.

  Patent Document 1 proposes a method of separating a foreground and a background using time-space information of a video and processing a signal generated on the background as noise. Patent Document 2 proposes a method for creating a spatiotemporal feature amount using temporal differentiation and spatial differentiation obtained from images of several consecutive frames. Patent Document 3 proposes a method for detecting an object using co-occurrence of luminance gradient features.

JP 9-81714 A JP 2010-92293 A JP 2009-301104 A

  By the way, a technique for representing an image using statistics of a large number of local features (for example, SIFT) extracted from the image is used for object recognition in the image or image clustering. A method of extending this method to a moving image and recognizing the motion of the subject based on local features acquired from the video is also conceivable. However, methods such as MoSIFT (Motion SIFT) and STIP (Space-Time Interest Points), which expand feature quantities used in still images in the time direction, do not have a sufficient amount of information on the time axis.

  When detecting a human motion from an image, it is necessary to handle relatively long time information (for example, several seconds) in addition to spatial information. In addition, human movements vary greatly from person to person, and how they appear and change greatly depending on the shooting direction. For these reasons, it has been difficult to describe feature quantities suitable for human motion analysis.

  In addition, in an image or video expression technique such as BoF, local feature position information is ignored in order to maintain invariance due to a size change. A technique such as Spatial-Pyramid has been devised to embed the position information of the local features in the space. However, since there is no information in the time direction, it has been difficult to detect the movement of the person from the video.

  An object of the present invention is to provide an identification device and an identification program capable of accurately identifying an operation indicated by an image.

  An identification apparatus according to the present invention is an apparatus for identifying an operation indicated by an image, wherein an image dividing unit that divides the image into spatiotemporal subregions, and a feature point in the region for each of the spatiotemporal subregions Calculating a local feature amount related to the trajectory of the feature, a feature description portion described in a predetermined dimension, a feature processing portion for statistically processing the set of local feature amounts to obtain a feature representation, and a feature representation for each of the spatio-temporal subregions And an identification unit that identifies the operation by a predetermined classifier based on the set.

  According to this configuration, the identification device creates a feature expression that takes position and time information into consideration using a small space-time region. As a result, the identification device is robust to temporal and spatial changes in operation, and can detect a specific operation performed by a person in the video content with high accuracy.

  The feature representation unit includes a partial feature representation obtained from the spatiotemporal subregion and a feature pair that includes the spatiotemporal subregion and combines the global feature representation obtained from the large region having the same time scale. The creating unit may identify the operation based on the set of feature pairs.

  According to this configuration, the identification device can improve the identification accuracy of the operation by generating the feature expression including the motion information of the camera in the large area using the feature pair.

  The video dividing unit may divide the video so as to include a plurality of spatio-temporal small regions that overlap each other and have different time scales.

  According to this configuration, by preparing a plurality of temporal scales in a small spatiotemporal region, the identification device can cope with individual differences in the operation speed and can improve the identification accuracy of the operation.

  The identification unit may perform a second identification process using a combination of results of the first identification process as an input, following the first identification process using each of the feature pairs as an input.

  According to this configuration, the identification device can reduce the number of dimensions in each stage by identifying the feature expression with the two-stage classifier, and can accurately identify the human action with a small number of dimensions.

  The feature description unit includes a motion feature indicating a direction and intensity distribution of a motion vector in a trajectory of a feature point, an appearance feature indicating a color distribution around the trajectory, and a shape indicating a time length and a moving distance of the trajectory. A feature amount representing a feature may be calculated.

  According to this configuration, the identification device can efficiently express the feature of the motion indicated by the video using the appearance feature, the motion feature, and the shape feature, and can improve the identification accuracy of the motion.

  An identification program according to the present invention is a program for causing a computer to identify an operation indicated by a video, and includes a video dividing step for dividing the video into a spatio-temporal subregion, and each of the spatiotemporal subregions Calculating a local feature amount related to a trajectory of a feature point within the feature point, describing a feature in a predetermined dimension, performing a statistical process on the set of local feature amounts, obtaining a feature representation, and each time-space subregion And an identification step of identifying the action by a predetermined classifier based on the set of feature expressions.

  According to the present invention, an operation indicated by an image can be identified with high accuracy.

It is a block diagram which shows the structure of the identification device which concerns on 1st Embodiment. It is a figure which shows the flow of the identification process which concerns on 1st Embodiment. It is a figure explaining the appearance characteristic which concerns on 1st Embodiment. It is a figure explaining the movement characteristic which concerns on 1st Embodiment. It is the schematic of the feature expression which concerns on 1st Embodiment. It is a figure which shows the flow of the identification process which concerns on 2nd Embodiment. It is a figure which shows the evaluation experiment result which compared the identification accuracy by 1st Embodiment and 2nd Embodiment with the conventional method.

[First Embodiment]
The first embodiment of the present invention will be described below.
The identification device 1 according to the present embodiment identifies an operation performed by a person or the like in a video. Here, the video is content expressed by a moving image, and includes, for example, broadcast video, home video, Internet video, and the like transmitted at a predetermined frame rate.

FIG. 1 is a block diagram showing the configuration of the identification device 1 according to this embodiment.
The identification device 1 includes a control unit 10, a storage unit 20, a communication unit 30, an input unit 40, and a display unit 50.

The control part 10 is a part which controls the whole identification apparatus 1, and implement | achieves the various functions in this embodiment by reading and executing the various programs memorize | stored in the memory | storage part 20 suitably. The control unit 10 may be a CPU (Central Processing Unit).
In addition, the control unit 10 includes a video dividing unit 11, a feature description unit 12, a feature expression unit 13, and an identification unit 14, and executes identification processing described later.

  The storage unit 20 is a storage area for various programs and various data for causing the hardware group to function as the identification device 1, and may be a ROM, a RAM, a flash memory, a hard disk (HDD), or the like. Specifically, the storage unit 20 stores, in addition to an identification program for causing the control unit 10 to execute each function of the present embodiment, a feature description, a feature expression, and learning data of the classifier described later.

  The communication unit 30 is a network adapter when the identification device 1 communicates with other devices. Specifically, the communication unit 30 communicates with a communication terminal or various servers via a network according to the control of the control unit 10. Thereby, transmission / reception of video data, learning data, identification result data, and the like is performed.

  The input unit 40 is an interface device that receives an instruction input from the user to the identification device 1. The input unit 40 includes, for example, a keyboard, a mouse, a touch panel, operation buttons, and the like.

  The display unit 50 displays a screen for accepting data input to the user or displays a processing result screen by the control unit 10 according to display control by the control unit 10. The display unit 50 may be a display device such as a liquid crystal display (LCD) or an organic EL display.

FIG. 2 is a diagram illustrating a flow of identification processing executed by each unit of the control unit 10 according to the present embodiment.
The identification process includes a video segmentation step (step S1) by the video segmentation unit 11, a local feature extraction step (step S2) and a feature description step (step S3) by the feature description unit 12, and a feature representation step (step S4) by the feature representation unit 13. ) And the identification step (step S5) by the identification unit 14 are executed in this order.
Hereinafter, the function of each part will be described in detail.

  In step S1, the video dividing unit 11 divides the video into space-time small areas. A spatio-temporal small region is a region obtained by dividing space and time into predetermined units. For example, in the case of a broadcast video, this is an area obtained by extracting a portion having a common coordinate from each of a predetermined number of consecutive frames.

In BoF, since one feature vector is created from the entire screen, the position information of the local feature extracted from the video is not considered. Further, since the time scale for extracting the trajectory is fixed, the change in the operation speed is not allowed. These problems are caused by the fact that the extraction range of trajectory features is fixed spatially and temporally.
Thus, in the present embodiment, feature expression spatial-temporal multiscale bags (stMB) that consider the position of local features in space and time are used. stMB defines a set of spatially and temporally divided subregions, and local features obtained from the inside of each of these subregions are treated as individual features to represent spatial and temporal position information. Can be embedded. This makes it possible to obtain a feature representation that is robust to temporal and spatial changes in motion and that has high motion identification performance based on video analysis.

Specifically, the video dividing unit 11 divides the video so as to include a plurality of spatio-temporal small regions that overlap each other and have different time scales.
For example, the video dividing unit 11 spatially classifies nine types of images (for example, three seconds vertically and horizontally), and three types temporally (1 second, 2 seconds) with respect to the identification target video (for example, video for 3 seconds). 3 seconds).

In step S <b> 2, the feature description unit 12 extracts a feature point in this space-time small region.
The feature points are a plurality of points that can be tracked over time, such as edges in the image, and are extracted by an existing image analysis method.

In step S <b> 3, the feature description unit 12 calculates a local feature amount related to the trajectory of the extracted feature point and describes it in a predetermined dimension. The trajectory of the feature point is a movement history of the feature vertex in the image, and has both spatial information and time information.
Specifically, the feature description unit 12 includes a “motion feature” indicating the direction and intensity distribution of the motion vector in the trajectory of the feature point, an “appearance feature” indicating the color distribution around the trajectory, and the trajectory time. A feature amount representing a “shape feature” indicating the length and the moving distance is calculated.

FIG. 3 is a diagram for explaining appearance features according to the present embodiment.
The feature description unit 12 applies the color of the points around the feature points (in the square region) to any region of the RGB color space divided into 27, and creates a 27-dimensional color histogram. Thereby, the color distribution around a certain feature point is expressed as a feature quantity of a predetermined dimension.

FIG. 4 is a diagram for explaining the movement feature according to the present embodiment.
The feature description unit 12 divides the trajectory of feature points into motion vectors between frames, performs quantization processing according to the direction and size of each vector, gives 25 different labels, and creates a 25-dimensional histogram To do. Thereby, the direction and size distribution of a certain feature point is expressed as a feature quantity of a predetermined dimension. At this time, since the information of all the frames in the trajectory is used, the lack of information effective for identifying the operation is suppressed.

In step S4, the feature expression unit 13 performs statistical processing on the set of local feature values obtained in step S3 to obtain a feature expression.
The feature expression unit 13 projects each local feature amount onto the feature space, and then integrates these into a fixed dimension feature vector. First, the feature expression unit 13 projects all local feature values onto the feature space, and performs clustering processing according to the feature value distribution. Thereafter, the feature expression unit 13 performs statistical processing such as quantization and averaging for each cluster, and expresses the image as one feature vector. Although the number of feature points extracted varies depending on the image, a set of feature amounts can always be converted into a fixed dimension feature vector through this step regardless of the number of feature points.

This step is often processed by BoF, but in this embodiment, the feature expression unit 13 uses a VLAD (Vector of Locally Aggregated Descriptors) technique.
Specifically, the feature representation unit 13 sets a set of local feature amounts obtained from a plurality of feature points in a region as a predetermined number of code words (for example, 16) × the number of dimensions of local feature amounts (27 of the appearance features). A partial feature representation is generated as (dimension + 25 dimensions of motion features + two dimensions of shape features) information.
The partial feature expression is calculated as a vector sum of local feature amounts classified into the code word from the center for each code word.

  Further, the feature expression unit 13 combines the partial feature expressions obtained from the spatio-temporal small regions by stMB to obtain the entire feature expression.

FIG. 5 is a schematic diagram of feature expression according to the present embodiment.
Since the feature representation unit 13 is provided with nine types of spatial scales and three types of temporal scales, the feature representation has a form in which 27 types of histograms are integrated in total. The dimension of each histogram differs depending on the description method of the feature amount, and is the number of codewords in BoF, and the number of codewords × number of feature dimensions in the VLAD of this embodiment.

As described above, the feature expression unit 13 can generate a feature amount considering both position and time by acquiring a trajectory feature for each small region in space-time using stMB.
Usually, a photographer often takes a picture of a subject in consideration of horizontal and vertical three-divided lines, which is referred to as a three-division method. Therefore, this spatial division method can be expected to be particularly effective for broadcast images taken by a professional cameraman. . In addition, the feature expression unit 13 can cope with individual differences in the operation speed by preparing a plurality of temporal scales of small areas.

In step S <b> 5, the identification unit 14 identifies an operation by a predetermined classifier based on a set of feature expressions for each space-time small area.
Specifically, the identification unit 14 receives the fixed-dimension vector created in step S4, and is one of a plurality of types of operations set in advance by a classifier such as a support vector machine (SVM). Judgment is made. Note that the discriminator learns in advance by “supervised learning” given correct data.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described.
In addition, about the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted or simplified.
The identification device 1a of the present embodiment is different from the first embodiment in the functions of the feature expression unit 13a and the identification unit 14a in the control unit 10a.

  The feature expression unit 13a creates a partial feature expression obtained from a spatio-temporal small region and a feature pair that includes the spatio-temporal small region and combines a global feature representation obtained from a large region having the same time scale. To do.

In general, there are some patterns in broadcast video camera work, and camera operations are often performed according to these patterns. This tendency is particularly noticeable in sports broadcast video. For example, in a baseball broadcast, a long shot video in which a pitcher is photographed from behind is used for a pitching scene, and when a hit occurs, the camera immediately switches to a camera that follows the ball. Golf putting starts with a long shot that captures the entire green and moves to a zoomed-in image of the ball as the ball moves. In basketball, since the attack moves to the opponent team after shooting, the camera is panned in the opposite direction. As described above, there is a typical pattern for camera operation using a human action as a key.
Local feature values generated by camera operation and obtained from the background area of the video are generally treated as noise, but these are also feature values that represent camera movement. Consideration is considered to contribute to the recognition accuracy of human motion.

Therefore, the feature expression unit 13a creates a co-occurrence feature amount between the spatial small area and the entire image, and expresses both the motion of the subject and the motion of the camera.
Specifically, the feature representation unit 13a creates a pair of a global feature representation (Global Representation) extracted from the entire screen and a partial feature representation (Local Representation) extracted from a small region, and this feature pair (GPR: Global Representation). Let Pairwise Representation) be the basic unit of the co-occurrence feature amount of the spatio-temporal small region and large region.

The identification unit 14a identifies an operation based on the set of feature pairs.
Specifically, the identification unit 14a performs a second identification process using the combination of the results of the first identification process as an input, following the first identification process using each feature pair as an input.

FIG. 6 is a diagram showing a flow of identification processing by the identification unit 14a according to the present embodiment.
In the first step, the identification unit 14a inputs the single feature pairs (GPR), the SVM regression, input feature pairs _{^(P 1} 1 _~P ^{NT NS)} each identified space region when corresponding to each The reliability including the target action (actions 1 to N _A ) is output as a continuous value (a real number from 0 to 1).

For example, when the number of space-time divisions (= number of feature pairs) is 27 (space division number N _S = 9, time division number N _T = 3) and there are N _A types of actions to be identified, this first step 27 × N _A SVM discriminators are required. Identifying unit 14a, after calculating the reliability in all SVM classifier combines these are expressed as a feature vector of 27 × N _A dimension.

In the second stage, the identification unit 14a, a feature vector generated in the first stage, and input to N _A class SVM classifier, which identifies which of the plurality of target operation. With this two-stage SVM, the identification unit 14a can identify a human action without sufficiently expanding the number of dimensions of the feature amount and sufficiently considering the feature amount extracted in each small region.

  In this way, 27 pairs of GPRs are generated from the 27 spatio-temporal regions formed by stMB. These can be simply connected to form one feature vector. However, in this case, since the dimensions are very high, a large amount of learning data is necessary for the discriminator to have sufficient accuracy. Therefore, the identification unit 14a uses two-stage SVM classifiers with different input feature quantities, and at each stage, performs classification processing with low-dimensional feature quantities and at the same time improves the final identification accuracy.

[Experimental result]
FIG. 7 is a diagram showing evaluation experiment results comparing the identification accuracy of the identification device 1 according to the first embodiment and the identification device 1a according to the second embodiment with a conventional method.
The results of comparing the existing three methods (BoF with the number of codewords k = 16, BoF with the number of codewords k = 1000, VLAD with the number of codewords k = 16) and the two new methods (stMB, GPR) are shown. The number of code words in the new two methods is 16.

Broadcast video data for two games was used for the evaluation. The identified actions are 4 types (jump, layup shoot, jump shoot, set shoot) and “no target action”. The classifier was learned using the video for one hour and evaluated in the remaining one hour. The feature amount obtained during the time from the start to the end of each operation was learned as a positive example, and the feature amount at the time when no operation occurred was learned as a negative example. The occurrence timing (correct data) of each operation was manually created by visual inspection.
The accuracy was evaluated by the F value (%), which is a harmonic average of the proportion of correct answers (acceptance rate) included in the identification result and the proportion of all correct answers (reproducibility).

When VLAD was described using stMB (first embodiment), higher identification accuracy (Total) was obtained than in the conventional three methods. Thereby, the effectiveness of the description method (first embodiment) in consideration of the position of the trajectory feature is shown. In particular, sports images tend to perform a specific action (for example, a shot) in a specific area in the screen, so that the effect of using position information is high. Furthermore, accuracy was improved by changing the scale in the time direction and absorbing individual differences in operating speed.
In the individual operation, the accuracy of the existing VLAD may be high, but high accuracy can be expected by selectively using the method with higher accuracy according to the identification target.

  Moreover, when it identified by 2 step | paragraph SVM by a feature pair (GPR) (2nd Embodiment), the precision further improved. In the basketball video used for the evaluation, a certain pattern was seen in the camera operation, such as panning the camera in the opposite direction immediately after the goal. Since GPR learns camera operation patterns using human actions as keys, high identification accuracy is obtained.

As described above, the identification device 1 creates a feature expression that takes position and time information into account using stMB. As a result, the identification device 1 is robust against temporal and spatial changes in operation, and can detect a specific operation performed by a person in the video content with high accuracy.
This method can be widely applied to all types of video such as broadcast video, video on the Internet, and monitoring video, and can be used for searching using behavior as a key or monitoring a specific operation. In addition, this method is particularly suitable for analysis of sports broadcast video in which many fixed patterns exist for camera operation.

  Since the identification device 1 can efficiently express the feature of the motion indicated by the video using the appearance feature, the motion feature, and the shape feature, the identification accuracy of the motion can be improved.

  The identification device 1 can cope with individual differences in operation speed by preparing a plurality of time scales in stMB, and can improve the identification accuracy of the operation.

  The identification device 1a can improve the identification accuracy of the operation by generating the feature expression including the motion information of the camera in the large area using the feature pair (GPR).

The identification device 1a is able to improve the detection accuracy of the human motion by identifying the feature expression combining stMB and GPR with a two-stage classifier.
Further, when the histograms obtained from all the regions are simply arranged, the final number of dimensions is enormous, for example, 54 locus feature dimension number × 16 codeword number × 9 space division number × 3 time division number = 23328. Therefore, sufficient accuracy cannot be expected with a small amount of learning data, but the human motion can be accurately identified with a small number of dimensions by this two-stage identification method.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. Further, the effects described in the present embodiment are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the present embodiment.

The number of space-time divisions in stMB can be set as appropriate.
The discriminator may be able to switch the discriminator group according to the target video (for example, the type of basketball, American football, etc.). In this case, a specific discriminating target operation is set for each discriminator. Is done.

  In the present embodiment, the configuration and operation of the identification device have been mainly described. However, the present invention is not limited to this, and each component may be provided as a method or program for identifying an operation in a video. Good.

  Furthermore, the program for realizing the function of the identification device may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed.

  The “computer system” here includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer system.

  Furthermore, “computer-readable recording medium” means that a program is dynamically held for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include one that holds a program for a certain time, such as a volatile memory inside a computer system that becomes a server or client in that case. Further, the program may be for realizing a part of the above-described functions, and may be capable of realizing the above-described functions in combination with a program already recorded in the computer system. .

DESCRIPTION OF SYMBOLS 1, 1a identification device 10, 10a Control part 11 Image | video division part 12 Feature description part 13, 13a Feature expression part 14, 14a Identification part

Claims (7)

An identification device for identifying an operation indicated by an image,
A video dividing unit for dividing the video into space-time sub-regions;
For each of the spatiotemporal subregions, calculate a local feature amount related to the trajectory of the feature point in the region, and describe a feature description unit with a predetermined dimension;
A feature expression unit for statistically processing the set of local feature values to obtain a feature expression;
An identification unit that identifies the action by a predetermined classifier based on a set of feature expressions for each of the spatio-temporal subregions ,
The image dividing section, overlap each other, and said you divide the image identification apparatus to include a spatial subregion when a plurality of time scales are different. An identification device for identifying an operation indicated by an image,
A video dividing unit for dividing the video into space-time sub-regions;
For each of the spatiotemporal subregions, calculate a local feature amount related to the trajectory of the feature point in the region, and describe a feature description unit with a predetermined dimension;
A feature expression unit for statistically processing the set of local feature values to obtain a feature expression;
An identification unit that identifies the action by a predetermined classifier based on a set of feature expressions for each of the spatio-temporal subregions,
The feature representation unit includes a partial feature representation obtained from the spatiotemporal subregion and a feature pair that includes the spatiotemporal subregion and combines the global feature representation obtained from the large region having the same time scale. make,
The identification unit, based on a set of the feature pairs, identification device that identifies the operation.   The identification device according to claim 2, wherein the video dividing unit divides the video so as to include a plurality of spatiotemporal small regions that overlap each other and have different time scales.   The said identification part performs the 2nd identification process which uses the combination of the result of the said 1st identification process as an input following the 1st identification process which inputs each said feature pair. The identification device described in 1.   The feature description unit includes a motion feature indicating a direction and intensity distribution of a motion vector in a trajectory of a feature point, an appearance feature indicating a color distribution around the trajectory, and a shape indicating a time length and a moving distance of the trajectory. The identification device according to claim 1, wherein a feature amount representing a feature is calculated. An identification program for causing a computer to identify an operation indicated by an image,
A video dividing step of dividing the video into space-time sub-regions;
For each of the space-time subregions, calculate a local feature amount related to the trajectory of the feature point in the region, and describe the feature description step in a predetermined dimension;
A feature expression step of statistically processing the set of local feature values to obtain a feature expression;
An identification step for identifying the action by a predetermined classifier based on a set of feature expressions for each of the spatio-temporal subregions ,
In the image segmentation step, mutually overlap, and, because of the identification program the to divide the video to include a spatial subregion when a plurality of time scales are different. An identification program for causing a computer to identify an operation indicated by an image,
A video dividing step of dividing the video into space-time sub-regions;
For each of the space-time subregions, calculate a local feature amount related to the trajectory of the feature point in the region, and describe the feature description step in a predetermined dimension
A feature expression step of statistically processing the set of local feature values to obtain a feature expression;
An identification step for identifying the action by a predetermined classifier based on a set of feature expressions for each of the spatio-temporal subregions,
In the feature representation step, a partial feature representation obtained from the spatiotemporal subregion and a feature pair including the spatiotemporal subregion and a global feature representation obtained from a large region having the same time scale are combined. Let them create
An identification program for identifying the operation based on the set of feature pairs in the identification step.

Images