JP5766623B2 - Motion analysis apparatus, method, and program - Google Patents

Description

  The present invention relates to a motion analysis device, method, and program, and more particularly, to a motion analysis device, method, and program for analyzing motions of an object, a human body, and the like based on appearance (appearance of an object / human body).

  Analysis of the motion of an object or a human body is extremely important in a technical field called computer vision. For this purpose, a method based on appearance is widely used. Among them, the technique based on Motion History Image (MHI) is very widely used due to the ease of mounting and the like (for example, see Non-Patent Document 1).

  MHI is an image representing the motion of an object, and as a pixel value of each pixel, a bright value is given if there is a motion in a near time, and it gradually becomes dark if there is no motion. Specifically, in a frame image (video) continuous in time series, a bright luminance value is given to a pixel corresponding to a portion in which a luminance change is recognized in a recent frame (frame to be processed), It is an image whose luminance value becomes darker as time passes. The degree of darkening is determined by the Decaying parameter.

  MHI is a technique that is resistant to noise. The operation to be searched is expressed by one MHI (template MHI), and the operation of the search target is made by matching the template MHI with the MHI created from the video to be checked. Attempts have been made to identify where it is occurring.

Bobick et al., "The recognition of human movement using temporal templates", IEEE TPAMI, 23 (3), pp.257-267, 2001.

  In the biomechanics field, the goal is to analyze a specific movement (for example, pitching form). For example, the time evolution of a state (posture, etc.) within a series of movements, and different trials of a series of movements In (Trial), there is a request to analyze the difference between positions where each posture is detected for each trial.

  However, in the conventional method, the MHI is a single image that holds a history of operation for a certain period of time determined by parameters, and therefore, the most similar time to the template MHI can be obtained by collation using the template MHI. Therefore, the time evolution of the state in motion, that is, it moved more slowly than in the previous trial, or moved slowly until a certain point in the motion, but then moved quickly, There is a problem that it is not possible to analyze how the operating state has transited in time. Similarly, there is a problem in that it is impossible to analyze what positional relationship the operating state occurs compared to the time of the previous trial.

  The present invention has been made in view of the above problems, and an object thereof is to provide an operation analysis apparatus, method, and program capable of analyzing a difference in motion between trials in different trials of the same operation. To do.

In order to achieve the above object, the motion analysis apparatus of the present invention provides an object or a human body in a collation video for each of the plurality of frame images based on a collation video composed of a plurality of frame images continuous in time series. Creation means for creating a Motion History Image (MHI) representing the movement history of each pixel by a change in the pixel value of each pixel as a matching MHI , each of the matching MHI created by the creating means for each frame image, the MHI for each frame created from template image representing the operation to be analyzed against a time series template MHI obtained by arranging in time series as a template MHI, the template MHI most similar to each of the matching MHI said what number of templates MHI information indicating whether the time series template MHI, and the matching MH Is configured to include a, a collating means for outputting as a position matching result of the template MHI in.

  According to the motion analysis apparatus of the present invention, the creation unit is configured to detect the motion of an object or a human body in the collation video for each of the plurality of frame images based on the collation video including a plurality of frame images continuous in time series. A Motion History Image (MHI) representing the history as a change in the pixel value of each pixel is created as a matching MHI. The pixel value can be, for example, a luminance value or color information. As a change in pixel value, for example, a pixel whose movement has recently been recognized gives a bright luminance value, and a pixel whose movement has not been recognized darkens its luminance value over time. Express the history of.

When the collation unit arranges each of the collation MHI created by the creation unit for each frame image and the MHI for each frame created from the template video representing the operation to be analyzed as a template MHI in time series Matching with the sequence template MHI, the information indicating which template MHI the template MHI most similar to each of the matching MHI is the template MHI of the time series template MHI , and the matching result of the position of the template MHI in the matching MHI Output as.

  Thus, in order to obtain the most similar template MHI and its position by collating the time-series template MHI in which the template MHI for each frame image is arranged in time series with the matching MHI created for each frame image, In different trials of the same operation, it is possible to analyze the difference in motion between trials such as temporal transition and positional relationship of the state during operation.

  Further, the matching MHI and the template MHI are pixels corresponding to a portion in which a pixel value of each pixel is expressed in a range from a first pixel value to a second pixel value and a motion is recognized in a frame image to be processed. The pixel value of the first pixel value, and the pixel value of the pixel corresponding to the portion in which no motion is recognized is gradually changed to the second pixel value over time, and the frame image A plurality of MHIs having different degrees of change to the second pixel value may be included.

  In this way, by creating a plurality of MHIs with different degrees of change in pixel values in the MHI for each frame image, an appropriate MHI corresponding to the case where the speed of movement changes during the operation can be obtained. Can be obtained.

Further, the motion analysis method of the present invention is a motion analysis method in a motion analysis apparatus including a creation means and a collation means, wherein the creation means generates a collation video composed of a plurality of frame images continuous in time series. Based on this, for each of the plurality of frame images, a Motion History Image (MHI) that represents a history of movement of an object or a human body in a matching video as a change in pixel value of each pixel is created as a matching MHI. A time-series template MHI in which each of the matching MHIs for each frame image created by the creating means and the MHI for each frame created from the template video representing the operation to be analyzed are arranged in time series as a template MHI; matches, the template MHI is the time series template MH most similar to each of the matching MHI What number of templates MHI information indicating whether, and a method for outputting the position of the template MHI in the collation MHI as the collation result.

  The behavior analysis program of the present invention is a program for causing a computer to function as each means constituting the behavior analysis device.

  As described above, according to the motion analysis apparatus, method, and program of the present invention, the time-series template MHI in which the template MHI for each frame image is arranged in time series, the matching MHI created for each frame image, In order to obtain the most similar template MHI and its position, it is possible to analyze the difference in motion between trials in different trials of the same operation.

It is a block diagram which shows the functional structure of the operation | movement analyzer of 1st and 2nd embodiment. It is a flowchart which shows the content of the template setting process routine in the operation | movement analysis apparatus of 1st Embodiment. It is a figure for demonstrating the MHI preparation method of each time in 1st Embodiment. It is a figure which shows an example of a time series template MHI sequence. It is a flowchart which shows the content of the template collation process routine in the operation | movement analysis apparatus of 1st Embodiment. It is a figure which shows an example of a collation result. It is a figure for demonstrating the difference in MHI by the difference in a Decaying parameter. It is a figure for demonstrating the multi time scale MHI preparation method of each time in 2nd Embodiment. It is a figure for demonstrating the other setting method of the information of a template area | region.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The motion analysis apparatus 10 according to the first embodiment stores a CPU (Central Processing Unit), a RAM (Random Access Memory), and a program for executing a template setting processing routine and a template matching processing routine described later. It is comprised with the computer provided with ROM (Read Only Memory). This computer can be functionally represented by a configuration including a template setting unit 20 and a template matching unit 40 as shown in FIG.

  The template setting unit 20 can be represented by a configuration including the setting unit 22. The template collation unit 40 can be represented by a configuration including an MHI creation unit 42 and a collation unit 44.

  The setting unit 22 receives a template video representing the operation to be analyzed, and creates a template MHI based on the MHI image corresponding to the region based on the template region information stored in the template region storage unit 24. And stored in the template storage unit 30. As will be described later, since the template MHI based on the MHI image at each time is created in time series, the plurality of templates MHI created in time series are hereinafter referred to as “time series template MHI sequences”.

  The template video that is input is a video that represents the operation to be analyzed (images of a plurality of frames that are continuous in time series), and may be input from a video file stored in advance in a storage device, or shot with a camera. The recorded video may be input. Each frame of the video is assigned a frame number t, which indicates that the frame with the frame number t is a frame at time t.

  The time-series template MHI sequence is created by creating MHI at each time from the frame images at each time in the template video and arranging them in time series. Specifically, the pixel value H (x, y, t) at the pixel (x, y) of MHI at time t is obtained by the following equation (1). Here, a case where a value of 0 (dark) to 255 (bright) is given as the pixel value of MHI will be described.

  Here, g is a Decaying parameter, which is a parameter indicating a degree of darkening when a luminance value of a pixel whose motion is not recognized is darkened over time. Here, since 256 gradations from 0 to 255 are used as the MHI pixel value, g = 4 can be said to be MHI composed of information for 256/4 = 64 frames. Similarly, when g = 16, the MHI is composed of information for 16 frames, and 64 is information for 4 frames.

D (x, y, t) is a function for determining whether an object or motion exists in the pixel (x, y) in the frame at time t in the input video. For example, threshold processing of background difference or interframe difference can be used,
D (x, y, t) = TH (I (x, y, t) −I (x, y, t−1))
Etc. can be used. Here, I (x, y, t) represents the pixel value of the pixel (x, y) in the frame at time t. TH (a) is a function for binarizing a by threshold processing. For example, if a is equal to or greater than the threshold, it is determined that an object or a motion exists in the pixel (x, y) and “1”. If a is less than the threshold, it is determined that there is no object or motion in the pixel (x, y), and “0” is returned. The threshold setting method is determined in advance. A constant threshold value can always be used, and a threshold value that maximizes the inter-class variance / intra-class variance may be set dynamically as in the binarization method of Otsu.

  The template area information includes a template creation start frame number and a template creation end frame number as information indicating a frame used for creating a time-series template MHI sequence in the input template video. Further, the information indicating the region to be cut out as a template includes the template position and size. Such template area information is stored in advance in the template area storage unit 24. The setting unit 22 cuts out the part indicated by the information in the template area from the created template MHI at each time, assigns a unique template number to each part (cut out template MHI), and sets the time-series template MHI. Stored in the template storage unit 30 as a sequence.

  Based on the search range information stored in the search range storage unit 46, the MHI creation unit 42 creates a verification MHI corresponding to the range and inputs the verification MHI every time t. Update.

  The collation video to be input may be input from a video file stored in advance in a storage device, or a video shot by a camera may be input. A frame number t is assigned to each frame of the video.

  Similar to the setting unit 22, the matching MHI at each time t is created or updated by obtaining the pixel value H (x, y, t) at the pixel (x, y) of the MHI using the equation (1).

  The search range information defines a range for collation with the template MHI from the created collation MHI, and includes a position and size indicating a (spatial) range in the image. The search range need not be rectangular. Further, when the entire matching MHI is used as the search range, it is not necessary to determine the search range in advance.

  The collation unit 44 collates the collation area in the search range with the time-series template MHI sequence stored in the template storage unit 30 using the collation MHI created by the MHI creation unit 42 and cut out in the search range. The template number of the most similar template MHI and the position of the collation area are output.

  More specifically, the matching MHI in the search frame is extracted as a matching area while scanning the search range with a search frame having the same size as the template MHI. The collation region at each extracted position is compared with each template MHI of the time-series template MHI sequence, and the template number of the most similar template MHI is obtained using, for example, the Euclidean distance between images. Further, the position of the search frame when the collation area is extracted is acquired. The position of the search frame can be, for example, the coordinate position (horizontal position and vertical position) of any corner or center of the search frame. The comparison between the template MHI and the collation area is not limited to the Euclidean distance between the images, and a conventionally known method can be applied as a method for comparing images and a distance scale.

  Next, a template setting processing routine executed in the motion analysis apparatus 10 of the first embodiment will be described with reference to FIG.

In step 100, information on the template area (template creation start frame number t _S , template creation end frame number t _E , template position and size) stored in the template area storage unit 24 is read.

Next, in step 102, one frame at time t (frame number t) is read from the template video in time series. Frame initially reads, from the frame of the frame number t _S a predetermined frame before (e.g., 1 frame before) the frame.

  Next, in step 104, a template MHI at time t is created. Specifically, as shown in FIG. 3, each pixel (x) in the frame at time t is obtained by taking the difference between the frame at time t (input image) and the frame at one time delay (t−1). , Y) is binarized to indicate whether an object or motion exists. Then, according to the formula (1), the pixel value of the pixel having the motion is set to 255, and the pixel value of the pixel determined to have no motion is darkened by g from the MHI pixel value of the frame delayed by one time. Create the MHI.

Next, in step 106, it is determined whether or not the frame number t of the frame read in step 102 is not less than the template creation start frame number t _S read in step 100 and not more than the template creation end frame number t _E. . If t _S ≦ t ≦ t _E, the process proceeds to step 108, and the template area having the template position and size indicated by the template area information read in step 100 is extracted from the template MHI created in step 104. Thus, a unique template number is assigned as the template MHI at time t and stored in the template storage unit 30. Then, the process returns to step 102, the next frame is read, and the process is repeated.

On the other hand, if it is determined in step 106 that t <t _S or t> t _E , the process proceeds to step 110 to determine whether or not the current frame number t exceeds the template creation end frame number t _E. To do. If t ≦ t _E , the process returns to step 102, the next frame is read, and the process is repeated. If t> t _E , the template setting process is terminated.

As a result, the template storage unit 30 stores a time-series template MHI sequence in which the templates MHI at each time t from t _S to t _E are arranged in time series. FIG. 4 shows an example of the time series template MHI sequence. The example of FIG. 4 is a time-series template MHI sequence related to the action of the pitcher throwing the ball, and the numerical value below each image indicates the template number.

  Next, with reference to FIG. 5, a template matching process routine executed in the motion analysis apparatus 10 according to the first embodiment will be described.

  In step 120, information on the search range (position and size indicating a (spatial) range in the image) stored in the search range storage unit 46 is read. Next, in step 122, one frame at time t (frame number t) is read from the verification video in time series.

  Next, in step 124, a matching MHI at time t is created as in step 104 of the template setting process. In the second and subsequent frames, the verification MHI created in the first frame is updated to obtain the verification MHI at time t.

  Next, in step 126, the search range read in step 120 is extracted from the matching MHI created or updated in step 124, and transferred to the subsequent process (step 130).

  Next, in step 128, it is determined whether or not the verification video has ended. If not, the process returns to step 122, the next frame is read, and the process is repeated. If the verification video is completed, the process proceeds to step 130.

  In step 130, the matching MHI in the search frame is extracted as a matching region while scanning each of the search ranges at each time t cut out in the matching MHI in step 126 with a search frame having the same size as the template MHI. Each collation area extracted at each position of the collation MHI is compared with each template MHI of the time-series template MHI sequence, and the template number of the most similar template MHI is obtained. Further, the position of the search frame when the collation area is extracted is acquired.

  Next, in step 132, the collation result in step 130 is output, and the template collation process is terminated.

  FIG. 6 shows an example of the collation result. The upper part of FIG. 6 is a video display for confirmation, and the lower part of FIG. 6 shows the collation result. The three figures at the bottom of FIG. 6 show, from the left, the template number of the template MHI most similar to the collation area, the horizontal position from which the collation area was extracted, and the vertical position. It is. As shown in FIG. 6, it is possible to visualize which template MHI in the time-series template MHI sequence is most similar, and at what time and in which place (position on the image) the state has appeared. More specifically, from the left diagram in FIG. 6, it is possible to analyze how the operating state transitions in time. From the middle diagram and the right diagram in FIG. 6, between different trials. It is possible to analyze the difference in the positional relationship between the two.

  As described above, according to the motion analysis apparatus of the first embodiment, a time-series template MHI sequence in which the templates MHI of each time are arranged in time series is created, and created at each time of the collation video. In order to obtain the template MHI most similar to the matching MHI and the position thereof, the difference in motion between trials can be analyzed in different trials of the same operation.

  Next, a second embodiment will be described. In addition, about the operation | movement analyzer of 2nd Embodiment, about the part similar to the operation | movement analyzer 10 of 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  As shown in FIG. 1, the motion analysis apparatus 210 according to the second embodiment includes a template setting unit 220 and a template matching unit 240, and the template setting unit 220 can be expressed by a configuration including a setting unit 222. Further, the template matching unit 240 can be represented by a configuration including an MHI creating unit 242 and a matching unit 244.

  Here, the principle of MHI used in the second embodiment will be described.

  MHI has different appropriate Decaying parameters depending on the speed of movement of the object. When the motion is fast, using a small Decaying parameter results in an MHI with extremely small time resolution. On the other hand, if a large Decaying parameter is used when the motion is slow, the motion becomes an MHI where almost no motion is detected. When this is expanded in time series as in the first embodiment, it may be difficult to obtain an appropriate MHI with a single Decaying parameter because the speed of movement changes during the operation.

  This example will be described with reference to FIG. FIG. 7 is an example of MHI representing a pitching operation, and shows a processing result for an image of 30 fps (frame / second). Specifically, each MHI is shown when Decaying parameter d (τ) = 4, 16, 64 at times 1, 2,..., 5,. . The upper part of FIG. 7 shows a frame image at that time in the video. For example, with the Decaying parameter d (τ) = 4, the MHI at times 2, 5, and 10 are very similar. When d (τ) = 64, a large difference can be confirmed in the MHI obtained at times 2, 5, and 10. However, when the MHI in the case of d (τ) = 64 and time 10 is seen, it is imagined that motion (pixels with bright pixel values) is hardly included and it is difficult to detect the motion.

  Therefore, the template MHI and the matching MHI at each time used in the first embodiment are expanded to a new MHI that is a set of a plurality of MHIs generated by a plurality of different Decaying parameters to solve the above problem. In the second embodiment, the extended MHI is referred to as “multi-time scale MHI”. Hereinafter, each part is explained in full detail.

  Similar to the setting unit 22 of the first embodiment, the setting unit 222 receives a template video and inputs a time-series template MHI sequence based on the template area information stored in the template area storage unit 24. create. However, the template MHI for each time is created as a multi-time scale MHI according to the above principle. Specifically, the multi-time scale MHI is set to mH (x, y, t) = {H1 (x, y, t),..., HN (x, y, t)}, and the following equation (2) Ask for.

  Here, N is a predetermined number of time scales, and d (1), d (2),..., D (N) are Decaying parameters in each time scale. The setting unit 222 cuts out the portion indicated by the template area information from the created template MHI for each time scale at each time, and assigns a unique template number to each portion (cutout template MHI). Stored in the template storage unit 30 as a sequence template MHI sequence.

  Similar to the MHI creation unit 42 according to the first embodiment, the MHI creation unit 242 receives an image for verification and corresponds to the range based on the search range information stored in the search range storage unit 46. The matching MHI is created, and the matching MHI is updated every time t. However, the MHI for collation at each time is created or updated as a multi-time scale MHI according to the equation (2) according to the above principle.

  Similar to the collation unit 44 of the first embodiment, the collation unit 244 receives the collation MHI created by the MHI creation unit 42 and cut out in the search range, and the collation area in the search range and the template storage unit 30 is collated with the time-series template MHI sequence stored in 30, and the template number of the most similar template MHI and the position of the collation area are output. When comparing the collation area and each template MHI, images having the same Decaying parameter are compared.

  The template setting process and the template matching process in the behavior analysis apparatus 210 of the second embodiment are replaced with the point that the MHI is created in the template setting process and the template matching process in the behavior analysis apparatus 10 of the first embodiment. Since only the point of creating the multi-time scale MHI is different, a method of creating the multi-time scale MHI will be described with reference to FIG.

  First, an object or a motion exists in each pixel (x, y) in the frame at time t by taking a difference between the frame at time t (input image) and the frame with one time delay (t−1). Binarization is performed to indicate whether or not. This corresponds to D (x, y, t). Then, according to equation (2), the pixel value of a pixel in which motion exists (pixel of D (x, y, t) = 1) is set to 255, and a pixel (D (x, MHI1 is created by darkening the pixel value of y, t) = 0) by d (1) from the MHI pixel value of the one-time-delayed frame created as Decaying parameter = d (1). In addition, the pixel value of a pixel in which motion exists (pixel of D (x, y, t) = 1) is set to 255, and a pixel determined to have no motion (D (x, y, t) = 0) MHI2 is created by darkening the pixel value of d) by d (2) from the MHI pixel value of the frame delayed by one time created with the Decaying parameter = d (2). Similarly, MHIN is created for each time scale. That is, N MHIs having different Decaying parameters at time t are created.

  As described above, according to the motion analysis processing device of the second embodiment, a plurality of time-series multi-time scales MHI are created as a time-series template MHI sequence, and each time of the collation video is displayed. In order to obtain the most similar template MHI and its position by creating a matching MHI using the multi-time scale MHI, it is possible to analyze the difference in motion between trials in different trials of the same operation in more detail. .

  In the first and second embodiments described above, the template area is cut out using the template area information stored in advance in the template area storage unit 24. However, the input template video is confirmed. However, the template creation start frame, end frame, template position, size, and the like may be designated by a mouse operation or the like. An example of this is shown in FIG.

  In the interface of FIG. 9, a template creation start button 52, a template creation end button 54, a video playback button 56, and a frame advance button 58 are arranged beside a video playback window 50 in which an input video is displayed. In such an interface, a video is displayed in the video playback window 50 by selecting the video playback button 56 using a mouse click, an arrow key on the keyboard, or the like. The frame creation start button 52 is selected, and the template creation start button 52 is selected when the template creation start frame is displayed while the frame is advanced frame by frame. While continuing to advance the video frame by frame, the portion to be cut out as the template MHI on the video playback window 50 by the operation of the mouse or the like in each frame, that is, the template position and size are designated. When the template creation end frame is displayed, the template creation end button 54 is selected. In this way, the template area information can be specified while reproducing and checking the video.

  In the above embodiment, the case where the history of movement is expressed by darkening each pixel of the MHI from the bright luminance value as time elapses has been described. However, the darkness value gradually becomes darker or a certain color ( For example, the movement history may be expressed by gradually changing from red to another color (for example, blue).

  Moreover, although the case where the template setting part and the template collation part were comprised with the same computer was demonstrated in the said embodiment, you may make it comprise with a separate computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  In addition, the above-described motion analysis apparatus includes a computer system, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

10, 210 Motion analysis device 20, 220 Template setting unit 22, 222 Setting unit 24 Template area storage unit 30 Template storage unit 40, 240 Template collation unit 42, 242 MHI creation unit 44, 244 Collation unit 46 Search range storage unit

Claims (4)

Based on a collation image made up of a plurality of frame images that are continuous in time series, for each of the plurality of frame images, a motion that represents the movement history of an object or a human body in the image for collation as a change in the pixel value of each pixel. Creating means for creating a History Image (MHI) as a matching MHI;
And each of the matching MHI for each frame image created by the creating means, and a time series template MHI which are arranged in time series MHI for each frame created from template image representing the operation to be analyzed as a template MHI Collation and information indicating the number of the template MHI of the time-series template MHI that is most similar to each of the collation MHIs , and the position of the template MHI in the collation MHI as a collation result Collation means to output;
A motion analysis device including:   In the matching MHI and the template MHI, the pixel value of each pixel is expressed in the range of the first pixel value to the second pixel value, and the pixel corresponding to the portion in which the motion is recognized in the frame image to be processed MHI in which the value is the first pixel value, and the pixel value of the pixel corresponding to the portion in which no motion is recognized is gradually changed to the second pixel value over time, for each frame image The motion analysis apparatus according to claim 1, further comprising a plurality of MHIs having different degrees of change to the second pixel value. A motion analysis method in a motion analysis apparatus including a creation unit and a verification unit,
The creation means, based on a collation video consisting of a plurality of frame images continuous in time series, for each of the plurality of frame images, the history of the movement of the object or human body in the collation video is the pixel value of each pixel Create a Motion History Image (MHI) expressed as a change as a matching MHI,
When the collating unit arranges each of the collating MHI created by the creating unit for each frame image and the MHI for each frame created from the template video representing the operation to be analyzed as a template MHI in time series The template MHI is collated, information indicating which template MHI of the time-series template MHI is the template MHI most similar to each of the collating MHI , and the template MHI in the collating MHI A motion analysis method that outputs the position as a verification result.   An operation analysis program for causing a computer to function as each means constituting the operation analysis apparatus according to claim 1.