JPH08214289A - Device and method for analyzing time series image - Google Patents

Abstract

PURPOSE: To effectively execute time series analysis by highly accurately analyze the movement of an object for a part of which analysis is impossible due to the influence of shield or the like by a conventional method by utilizing knowledge relating to the shape and operation of the object. CONSTITUTION: An input part 1 executes photoelectric conversion for a photographed image and a video signal for prescribed time is transferred to an A/D conversion part 2 based upon the control of a control part 11. The A/D conversion part 2 converts the video signal into a digital image signal and transfers the digital image signal to a buffer memory part 3. The memory part 3 temporarily stores the transferred signal and outputs the stored data to an auxiliary I/O part 4 such as a magnetic disk device as the image data of each frame. The I/O part 4 outputs image data to a shape generating part 5 under the control of the control part 11. The generating part 5 calculates the absolute value of a difference between a background image in which analytical processing does not exist and an original image in which an object exists, binarizes the absolute value and extracts only a moving image area.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a time-series image analysis apparatus for extracting a predetermined object from a time-series image and analyzing it, and a time-series image analysis for highly accurately analyzing the operation of the object. The present invention relates to a device and an analysis method thereof.

[0002]

2. Description of the Related Art Generally, various techniques are known for recognizing an image captured by an image capturing device such as a CCD camera. For example, "description and representation of silhouette figures by ellipses" (Ihiko Kimoto et al. IEICE Transactions
D-II Vol. J76-DII No. 6 P115
9-P1167), 2 from the natural image.
By performing the value conversion process, the silhouette is extracted,
A full search is performed in the extracted silhouette area, and an ellipse with the largest area is fitted. Then, for the remaining silhouette areas that have not been fitted, elliptical fitting is similarly performed to recognize the shape of the object.

[0003]

However, in the above-mentioned conventional method, the object is recognized only by one still image and the silhouette image extracted from this still image.
Therefore, if the information on the original shape of the object is missing due to the influence of shadows or occlusions, it is possible to analyze which part is missing or what part is moving. Can not. Even if misrecognition occurs,
It cannot be determined if it is, in fact, a misrecognition.

Therefore, an object of the present invention is to provide a time-series image analysis apparatus and an analysis method thereof, which can analyze the movement of an object with high accuracy by utilizing knowledge about the shape and motion of the object. To do.

[0005]

In order to achieve the above object, in a time-series image analysis device for analyzing a predetermined object in a time-series image, a plurality of time-series images in a predetermined time range are analyzed. Shape generation means for extracting the object from the image,
Object model means for storing knowledge about the shape and shape change due to movement of the object as a parameter for each part constituting the object, and the object model from the object extracted by the shape generation means Parameter calculation means for calculating the position and inclination angle parameters of the part corresponding to the parameters stored in the means over time, and the object calculated by the parameter calculation means corresponding to each of the plurality of images Time-series image composed of matching means for matching a set of parameters having parameters as constituent elements with a set of parameters having respective parameters stored in the model means corresponding to the parameters of the object as constituent elements Provide an analysis device.

Further, in a moving image analysis method for analyzing a predetermined object in a time-series image, knowledge of the shape of the object and its shape change due to movement is parameterized for each part constituting the object. Modeled by storing the knowledge of motion as a function of time among the modeled parameters, extracting the object from a plurality of images within a predetermined time range in the time series image, and Calculating a parameter corresponding to the parameterized, synchronized between the modeled parameter and the parameter calculated from the object,
The modeled parameters are corrected so that the shapes of the parameters synchronized from the object and the shapes of the modeled parameters are matched, and the parameters calculated based on the object are used as the function. A time-series image analysis method that matches the parameters obtained from the above.

[0007]

With the above-described time-series image analysis device, an object is extracted from a plurality of images within a predetermined time range in the time-series image, and the shape and movement of the parts constituting the object are used as parameters. On the other hand, the model of the object is stored as a parameter corresponding to the above parameter. In the plurality of images as a whole, the parameters obtained from the object are set so that the parameters obtained from the object and the parameters of the model match, that is, the values of the respective parameters satisfy a predetermined relationship. By the correction, the movement of the object is recognized and analyzed with high accuracy. Alternatively, the parameters of the object are corrected so that an image configured as an image representing the object from the parameters calculated by the parameter calculation means and the object extracted by the shape generation means satisfy a predetermined relationship. To analyze the movement of the object.

Further, the time-series image analysis method is used to calculate the parameters of each part constituting the object in the time-series image, and the size of the model is corrected based on the extracted object. After the movements of the object and the model are synchronized, the parameters calculated from the object are corrected so that the magnitudes of the parameters calculated from the object and the parameters of the model satisfy a predetermined relationship.

[0009]

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 is a diagram showing a schematic overall configuration example of a time-series image analysis apparatus according to a first embodiment of the present invention.

This time-series image analysis apparatus comprises an image input section 1 including a CCD camera, an A / D section 2 for digitizing an image input signal from the input section 1, and an A / D section 2. A buffer memory unit 3 for temporarily storing the digital image signal of the frame unit, and the buffer memory unit 3
Auxiliary input / output unit 4 composed of a magnetic disk or the like for storing image data from the image data, and a desired object from the image data.
A shape generation unit 5 that extracts a (moving image area) and generates a silhouette image, and a silhouette image generated by the shape generation unit 5 correspond to the shape of each part (part) of the object to be analyzed in advance. A simple geometrical shape, for example, a model shape for expressing each part in a geometrical shape such as an ellipse shown in FIG. It is calculated by the parameter calculation unit 7 that divides the object into parts that form the object by the generation unit 5 and the object model unit 6 and roughly calculates the parameters of the position and tilt angle of each part, and the parameter calculation unit 7. A matching unit 8 for matching the entire parameter group with the entire parameter group output by the object model unit 6 and the silhouette image group generated by the shape generating unit 5, and the matching unit 8
The parameter group calculated in step 3 is output to the image generator 12. The D / A unit 9 for analogizing the output of the analysis result imaged by the image generating unit 12, the output unit 10 for outputting the analogized image data, and the control for controlling the operation of these components. And part 11. Image analysis in the time-series image analysis device configured as described above will be described.

First, the input unit 1 photoelectrically converts a captured image and transfers it to the A / D unit 2 as a video signal for a predetermined time under the control of the control unit 11. The A / D section 2 converts the video signal into a digital image signal and transfers it to the buffer memory section 3. The buffer memory unit 3 is composed of a plurality of frame memories, temporarily stores the digital image signal from the A / D unit 2, converts it into image data in frame units, and sequentially assists a magnetic disk device or the like. Output to the input / output unit 4.

The auxiliary input / output unit 4 outputs the image data to the shape generation unit 5 under the control of the control unit 11. The shape generation unit 5 calculates the absolute value of the difference between the background image in which the object of analysis processing does not exist and the original image in which the object exists in the image data, and then performs binarization processing to generate a moving image. Only the image area (silhouette image of the object) is taken out. Here, as the background image, a previously captured image may be used, or a background image may be generated from an original image and used.

The silhouette image generated by the shape generation unit 5 is output to the object model unit 6, the parameter calculation unit 7, and the matching unit 8, respectively.

Under the control of the control section 11, the object model section 6 calibrates the model size using the silhouette image and the information on the shape of the object. The parameter calculation unit 7
Then, under the control of the control unit 11, the parameters of the position and the inclination angle of each part of the target object are roughly determined from the silhouette image of the shape generation unit 5 and each output of the target object model unit 6.
As a specific example, a skeleton is extracted from the silhouette image generated by the shape generator 5. The extracted skeleton contains the width information of the original silhouette. Regarding the skeleton extraction method, for example,
It is described in Souken Shuppan, Computer Image Processing Primer (p77), etc., and detailed description is omitted here.

Next, the skeleton is divided based on the size of each part of the object model section 6. The center point of the divided skeleton corresponds to the position of each part, and the tilt corresponds to the tilt angle. Further, the width information of the skeleton at the center point is set as the allowable range ε.

The silhouette image generated by the shape generation unit 5, the model of the object model unit 6, and the parameters of each part calculated by the parameter calculation unit 7 are output to the matching unit 8. In the matching unit 8, the shape generation unit 5,
The outputs of the object model unit 6 and the parameter calculation unit 7 are used to synchronize the time regarding the operation of the object in the time-series image and the object model unit 6, and the entire parameter group calculated by the parameter calculation unit 7. And the relationship between the entire parameter group of the object model unit 6 and the silhouette image group of the shape generation unit 5 are collated to minimize the error.

The output of the matching unit 8 is transferred to the image generation unit 12 under the control of the control unit 11, and the calculated parameter is represented by, for example, an ellipse to form an image. The image generated by the image generation unit 12 is transferred to the D / A unit 9 and converted into an analog signal. Furthermore, the control unit 11
The above analog signal is output by the output unit 10 such as a display.
Output to and display the image analysis results.

Next, FIG. 2 is a diagram for explaining the model stored in the object model unit 6.

FIG. 2A shows a model in which, for example, a human being is an object, and this shape is decomposed into a plurality of parts constituting the object such as a body, head, and limbs, and is represented by an ellipse. . However, the above ellipse is an example of a simple geometric shape,
The numbers are the numbers of the parts, and a and b represent the length of each part. In this model, standard information that is known in advance regarding the object such as the size ratio of each part and the connection relationship is stored.

FIG. 2B shows the connection relationship and connection number of each part.

The size of each part of the object is defined as a ratio to the height of the object and stored in the object model unit 6. Further, in order to calibrate the size of each part, the size (height) mh in the height direction of the object of the model is also stored as a part of the model. Note that the simple geometrical shape is not limited to an ellipse, and it is of course possible to describe it with a line segment or a cylinder.

Further, theoretical numerical data of the position and inclination angle of each part of the target motion are stored corresponding to the time. In order to calculate the numerical data of the target motion of the object, the numerical data is collected using the markers. Here, the numerical data is the center position (x, y) of each part and the inclination angle θ.

A marker is attached to the center position of each part of the target object, and the target motion is photographed. A marker is extracted from the captured image, and the center of gravity of the extracted marker is set as the center point (x, y) of each part. For the inclination angle of this marker, for example, the extracted marker is thinned, and the inclination of this line is taken as the angle θ. The shape of the marker may be a shape such as a rectangle that allows calculation of the tilt angle.

The marker is a connection position of each part, that is,
It may be attached to the joint. In the numerical data calculation method in this case, markers are extracted and the positions of the markers are calculated. Using the positions of the two markers in the connected parts (1)
As shown in the formula, the length of the part, the inclination angle, and the center point are calculated.

By mounting this marker at the connecting position of each part, the length of each part can be accurately calculated. It should be noted that the marker shape need not be a circle shape as long as it does not hinder the operation and can calculate the length, inclination angle, and center point of each part. Further, the above two markers can be performed by using known markers such as color markers and light bulbs.

FIG. 13 is a conceptual diagram of the marker attachment and the numerical data calculation method. FIG. 13A shows an example in which the marker is attached to the central position of each part, and FIG. 13B shows the connection of the marker to each part. It is an example of being attached to a related part.

[0028]

[Equation 1]

Here, (xi, yi), (xj, yj)
Is the position of the marker of the part having the connection relation, and (x,
y) is the center point of each part, θ is the inclination angle formed by each part,
a represents the length of each part.

Further, all the calculated numerical data may be stored, or if the target motion has periodicity, the cyclic component of the motion may be extracted and stored for only one cycle.
The theoretical numerical data of the target motion may be stored as a table as shown in FIG. 14 corresponding to time, or each numerical data may be mathematically expressed by an interpolation function for each part and only the coefficient may be stored. May be saved. Further, it may be stored as a possible area as shown in FIG.

Next, FIG. 15 shows a concept of storing each numerical data stored in the object model unit 6 as a mathematical expression by an interpolation function for each site. As shown in FIG. 15A, a differential value (difference value) is calculated from the stored numerical data at time points i-1, i, i + 1.

Then, as shown in FIG. 15B, points at which the variation of the differential value becomes equal to or more than a predetermined threshold value are set as nodes. The extracted nodes are interpolated with a spline function or the like, and only the calculated coefficients are stored. The spline function is described in, for example, "Introduction to Spline Function" (Denki University Press). The nodes are at time i-1,
It may be an inflection point of the second-order differential value of each numerical data of i and i + 1. By using the inflection point as a node, it is possible to extract the node even for a motion in which the numerical data change is small.

When the motion of the target part has a periodicity, the cyclic component of the motion may be extracted and the logical (theoretical) numerical data of each part for one cycle may be stored. . Further, the logical (theoretical) numerical data of the operation of the target part may be stored in the form of a table corresponding to time. It may also be stored as a possible area as shown in FIG.

The possible region of each part in one cycle of walking as shown in FIG. 3B is calculated as follows based on theoretical numerical data.

A rectangle including the area (silhouette) of the object is extracted from the image of the model by a known method as shown in FIG. 3A, and the center line (midpoint) of this rectangle in the x-axis direction is extracted. Ask. The theoretical numerical data (center points) of the respective parts are superposed so that the center lines coincide with each other, that is, the x coordinates coincide with each other.

As a result, the distribution of the center points of the respective parts shown by the black dots in FIG. 3B can be obtained. An area that includes the distribution of the center points, that is, a possible area is defined as an area where the center points of the respective parts can exist. This possible area is described by a combination of parameters such as an ellipse and a semicircle that are geometric shapes.

Specifically, the head and the torso are indicated by ellipses centered on the average value of the center points, and the hand, upper arm, and forearm indicate the shoulder position, and the thigh, the knee, and the foot. The parts are shown as a combination of circles (fan-shaped) centered on the waist position. Shoulder / Waist position (xshoulder, yshoulder), (xwaist, ywaist)
Is calculated by the equation (2).

[0038]

[Equation 2]

Here, (x2, y2) is the center point of the body portion calculated above, a2 is the length of the body portion, and θ2 is the inclination angle of the body portion.

Using the equation (3), a geometrical shape including the center points of the hand, the upper arm, the forearm, the thigh, the knee, and the foot is calculated.

[0041]

(Equation 3)

Here, r is the radius of a circle that includes all the center points of the respective parts, and phi is the circumferential angle. 16
It is explanatory drawing of the calculation method of the radius r and the circumference angle phi. For these, the distance and angle between the center point ((xshoulder, yshoulder) or (xwaist, ywaist)) of the circle and all the center points of each part are calculated, for example, as shown in equation (4).

[0043]

[Equation 4]

Here, x03 is x at time 0 and part number 3
The coordinates are shown, and r0 and phi0 are the part number 3 from the waist position.
Shows the distance and angle to the thigh. This calculation is performed for all times of analysis, and the maximum and minimum values of r i and p h i are calculated from r max, r min, p i max, p h min
Can be automatically calculated. This operation is performed for each part.

The obtained possible existence area shown in FIG. 3 (b) is not stored as an image, but the radius r and the circumferential angle ph are stored.
Parameters such as i, center point (x, y), rotation angle θ, and major axis / minor axis lengths a and b are stored as a table. When processing is performed, the image is restored as an image shown in FIG. 3B using this parameter.

FIG. 4 is a block diagram showing a specific configuration of the matching section 8 described above.

In the matching section 8, the outputs of the shape generating section 5 and the object model section 6 are transferred to the synchronizing section 21. The synchronization unit 21 calculates a plurality of times at which the same operation is performed in order to synchronize with the silhouette image group of the shape generation unit 5 and the model parameter group of the object model unit 6. The outputs of the object model unit 6 and the synchronization unit 21 are transferred to the period calibration unit 22. In the present embodiment, it is premised that the operation to be folded has periodicity. Therefore, in the period calibrating unit 22, the time required for one period calculated from the plurality of synchronization times calculated by the synchronizing unit 21 and the model included in the model parameter output from the object model unit 6 in advance are calculated. 1
The cycle of model data is calibrated based on the time taken for the cycle. The shape generation unit 5, the object model unit 6, the parameter calculation unit 7, the synchronization unit 21, and the period calibration unit 22 are output to the parameter correction unit 23.

The parameter correction unit 23 corrects the parameters of the time-series image by comparing with the model in one image. The outputs of the shape generation unit 5, the object model unit 6, the parameter calculation unit 7, and the parameter correction unit 23 are transferred to the optimization processing unit 24. In the optimization processing unit 24, the entire parameter group calculated by the parameter calculation unit 7 is matched with the entire parameter group of the object model unit 6 and the silhouette image group of the shape generation unit 5 which are periodically calibrated.

FIG. 5 is a conceptual diagram of the processing of the synchronization unit 21. The synchronization unit 21 generates an ellipse as an example of a simple geometrical shape using the parameters of the model position and the tilt angle at the time time, and performs matching with the silhouette image group of the target object in the time-series image. The time of the silhouette image at which the matching error becomes the minimum corresponds to the time time in the model.

A concrete example of the synchronizing unit 21 when the object is a human will be described with reference to the flowchart shown in FIG.

First, from the model, the reference motion, "close", is generated.
The time cross in the state e (the state where the limbs are closed) is calculated (step S1). FIG. 18 is a conceptual diagram of a method of calculating the close state. The method of calculating the time of the reference motion close state is to calculate the parameters of the model stored in the object model unit 6 from the spline function, and calculate the head (part number 0) and other part groups at all times. x
Calculate the sum of the coordinate differences. The time when this sum becomes the smallest is cross. FIG. 5 is a conceptual diagram of a method of calculating the synchronization time time.

Next, the ellipse image of the model at time cross is generated using the parameters of the model at time cross (step S2). A circumscribed quadrilateral including the silhouette image of the model at the time cross and the silhouette image of the time series image at the time time is extracted by a known method,
The center lines of the circumscribed quadrilateral in the x-axis direction are made to coincide with each other. The area difference between the silhouette image of the time series image and the ellipse image of the model is calculated.

The area differences between the time-series images at all times and the silhouette image are calculated, and the time peri at which the area difference is the minimum is calculated.
This is the time when od performs the same operation as the time cross (step S3).

As described above, the model and the time-series image can be synchronized by matching the close-state elliptic image of the model with the silhouette image group of the object in the time-series image.

The synchronization time can also be calculated by the following method. A plurality of times at which the area of the silhouette image group of the model stored in the object model unit 6 and the area of the silhouette image group of the time-series images are minimum (or maximum) are calculated. The calculated time is the time of synchronization between the model and the time series image. For example, when human walking is analyzed, when the area of the silhouette image is the minimum,
It shows a nearly upright state. Here, the calculation of the minimum area time of the model and the silhouette of the time-series image means that the time in the upright state is synchronized.
By doing so, it is not necessary to generate an elliptical image, and thus the processing time can be shortened.

6 and 20 show the concept of the process of the period calibrating unit 22. The period calibration unit 22 is performed only when the target operation has periodicity. The time T required for one cycle is calculated based on the plurality of synchronization times calculated by the synchronization unit 21. The ratio ra = MT / T between the time MT required for one cycle of the model stored in the object model unit 6 and the time T required for one cycle of the time series image is calculated, and this ratio ra is obtained.
Calibrate the cycle of the model using.

Referring to the flow chart shown in FIG.
The period calibration unit 22 will be described. As the time MT of one cycle of the model stored in the object model unit 6,
According to the plurality of synchronization times calculated by the synchronization unit 21,
The time T required for one cycle is calculated (step S1
1). A ratio ra = MT / T between the time MT required for one cycle of the model and the time T required for one cycle of the time series image is obtained (step S12), and the parameter correction unit 23 calibrates the cycle of the model using this ratio ra. I do. This process is performed on a cycle-by-cycle basis. Therefore, it is possible to deal with a situation in which the motion of the object accelerates as the cycle increases. This is because the numerical data of the motion is stored as the coefficient of the spline function, so that the numerical data at any time can be calculated.

FIG. 7 is a conceptual diagram in the processing of the parameter correction unit 23. In the parameter correction unit 23, the parameters of each part at the time time of the time-series image and the time tim.
The difference between the parameters of each part of the model corresponding to e is calculated. The allowable range ε calculated by the parameter calculation unit 7 is used as a threshold. When the difference between each parameter is equal to or greater than a predetermined threshold value, the parameters of the time series image are replaced with the parameters of the model, and for example, an ellipse is generated as an example of a simple geometrical shape, and the error with the silhouette image is calculated. To do. If the calculated error is greater than before replacement, the parameter replacement is discarded. Repeat the above process for each parameter,
Modify the parameters in the image.

Referring to the flow chart shown in FIG. 21,
The parameter correction unit 23 will be described.

The size ratio of the object of the model and the object of the time series image is calculated (step S21). The size in the height direction of the object of the model stored in the object model unit 6 is mh, and the circumscribed quadrilateral including the object (silhouette image) of the time series image is calculated, and the size in the height direction is calculated. Calculate h. Using this, the ratio of the sizes of the model and the object of the time series image, scale = mh / h, is calculated. After the center position of each part of the object is multiplied by the ratio scale to correct the position, the following processing is performed.

The time of the model corresponding to the time series image is calculated using the ratio ra calculated by the period calibrating unit 22 (step S22). The time of the model is multiplied by the ratio ra, and if the time of the time series image exceeds the time of one cycle of the model multiplied by ra, the time is adjusted so as to correspond to the start time of the model.

Initialization of time time is performed (step S23). When the time is synchronized, an elliptic image is generated from the parameters of the time-series image at time time (step S).
24). The area difference between the silhouette image and the elliptical image of the time-series image at the time time is calculated and stored in diff1 (step S25).

The parameters of the model at the time corresponding to the time time at which the synchronization processing is performed are calculated from the spline function, and the absolute value of the difference between the parameters of the time model corresponding to the time time and the parameters of the time series image is calculated for each part. (Step S26). The absolute value of the difference is compared with the allowable range ε calculated by the parameter calculation unit 7 (step S27), and if the difference is small (NO), the process proceeds to step S28, and time is time.
Is incremented and the process returns to step S24.

However, if larger in step S27 (Y
ES), the parameters of the model and the time series image are exchanged for each part (step S29). Since the positions of the target object of the time-series image and the target object of the model are different, the silhouette image is used to perform the position adjustment and the parameter replacement is directly performed. A circumscribed quadrilateral including the silhouette image of the time-series image at time time and the silhouette image of the model corresponding thereto is extracted, and the center lines of the circumscribed quadrilateral in the x-axis direction are matched. That is, the amount of movement of the model for matching the x-coordinate of the center of the circumscribed quadrangle is calculated, and the parameters of the model are corrected and replaced based on this amount of movement.

An elliptic image is generated from the replaced parameters (step S30), and the area difference between the elliptic image and the silhouette image generated in step S30 is calculated as in step S25 and stored in diff2 (step S3).
1). The area difference diff1 before the parameter replacement is compared with the area difference diff2 after the parameter replacement (step S3).
2). If diff2 is greater than diff1 (N
O), the process returns to step S28, the time time is incremented, and the process returns to step S24. If small (Y
ES), discard the exchange of parameters (step S
33). In steps S24 to S33, all the parts are collectively processed. The time time is compared with the end time of the time series image, and when the time time is smaller than the end time (NO), the process returns to step S28 and the time ti
The me is incremented, and the process returns to step S24.
If it is larger (YES), the process ends.

FIG. 8 is a conceptual diagram of the processing of the optimization processing section 24. The parameter group of the time-series image calculated by the parameter calculation unit 7, the parameter group of the model of the object model unit 6 and the silhouette image group generated by the shape generation unit 5 are matched. As an example of a method for this, an evaluation function is set and minimized. For example, in the case of analysis targeting human beings, the evaluation function ET is defined as shown in equation (5), and the evaluation function ET is minimized by using annealing or the steepest descent method. Various parameters are calculated.

[0067]

(Equation 5)

Here, T is a cycle, i = T to T + 1 is time, j = 1 to 14 is a part, l = 1 to 13 is a connection number of each part as shown in FIG. 2B, and Pm is a model. , P is a parameter group of each part of the object in the time-series image. The first term matches the parameters of the object with those of the model. Sil is a silhouette image of the object in the time series image, Par is an elliptical image generated from the parameters of the time series image, and the second term matches the parameters of the time series image with the silhouette image.

Lm and L are 13 shown in FIG. 2 (b).
It is the distance between the center points of the parts having the connection relation of the set. In other words, it is the distance between the center positions of the parts such as the head part and the body part that are in a connection relationship. Lm represents the distance between the center points of the respective parts related to the model, and L represents the distance between the center points of the respective parts related to the time series image. The third term matches the connection relationship of the parameters between the respective parts of the object in the time-series image with the connection relationship of the model. k is the weight of each term.

It is also possible to calculate the optimum parameters of the parameter group of the time-series image by using the evaluation function ET as in the equation (6) by annealing or the steepest descent method.

[0071]

(Equation 6)

Here, T is a cycle, i = T to T + 1 is time, j = 1 to 14 is a part, and l = 1 to 13 is a connection number of each part as shown in FIG. 2B. Pm is a parameter group of the model, and P is a parameter group of each part of the object in the time series image. The first term matches the parameters of the object with those of the model. Sil is a silhouette image of the object in the time-series images, Par is an elliptical image generated from the model parameters, and the second term matches the model parameters to the silhouette images. Lm and L are shown in FIG.
It is the distance between the center points in the 13 sets of connection relationships shown in (b). In other words, it is the distance between the center positions of the parts having a coupling relationship such as the head and the body. Lm represents the distance between the center points of the respective parts related to the model, and L represents the distance between the center points of the respective parts related to the time series image. The third term matches the connection relationship of the parameters between the respective parts of the object in the time-series image with the connection relationship of the model. Pc is time i-2, i-1,
The weights of 1: 2: 3: 2: 1 are respectively given to the parameters of the time series images of i, i + 1, and i + 2, and the average value is calculated. Match the parameter value estimated from the time. k
Is the weight for each term.

A method for calculating the optimum parameters of the parameter group of the time-series image by utilizing the annealing by matching the above three parties will be described below. The optimization processing unit 24 will be described with reference to the flowchart shown in FIG.

First, the temperature Tempint and the number of iterations n are initialized (step S41). iteration
And Temp are initialized (step S42). Initial energy init eng is calculated (step S
43).

The initial energy represents the result of calculating the evaluation function of the equations (5) and (6). A random number ran is generated to determine a value g that gives a minute change, a parameter type xyr, a time time, and a site number p (step S4).
4). For example, each parameter when a random number from 0 to 1 is generated is: time = ran × number of images xyr = ran × 3 (xxy = 1, x, xyr = 2)
When y, xyr = 3, rad) p = ran × 14 g = length of site of site number p × (ran × 2.1-1.
0) is given.

Parameter x of the time-series image at time time
A small change g is given to yr [time] [p] (step S45). The energy eng is calculated using the parameter that has been slightly changed (step S46). Initial energy init eng and energy eng are compared (step S47), and init If eng is larger than eng (NO), the process proceeds to step S48 and the random number r
an and exp [(eng-init eng) / Tem
p] is compared, and if the condition is not satisfied (NO), the process returns to step S44 and a new random number ran is generated.

If the condition of step S48 or step S47 is satisfied (YES), this minute change is accepted, the parameter value is updated, and the energy value is exchanged (init). eng = eng) (step S
49).

It is judged whether or not the iteration count iteration is smaller than the initial value n (step S50), and if the condition is satisfied (YES), the temperature is lowered and the iteration count iterat is reached.
After updating ion (step S51), step S4
Returning to 4, a new random number ran is generated. However, if the condition is not satisfied (NO), the optimization process ends.

In performing the optimization process, the originally connected parts, for example, the hand part and the forearm part may be separated from each other depending on the conditions. In order to prevent this, a method of limiting the range of minute changes given to the parameters of each part and preventing separation of each part will be described. In order to prevent the separation of each part, a model that adaptively adjusts the length of the long axis and the short axis, which has been invariant in the past, is added. Specifically, a limitation is set to prevent separation when a small change is given in the optimization process.

Consider a case where a slight change is given to the center coordinates of each part.

At this stage, it is assumed that the long axes of a predetermined length are connected. The length of the major axis of the site of interest that gives a minute change and the site connected to it is extended by a predetermined length. Consider a circle whose radius is the extended major axis. Move the circle of the part of interest so that the circles do not separate, that is, always touch or intersect.

The locus drawn by the central coordinates at this time is an area where a minute change can be given while maintaining the coupling. Figure 2
FIG. 3 is a conceptual diagram regarding prevention of separation of each part when the object is a human. A case of a human will be specifically described based on FIG. For example, consider a case where a minute change is given to the center coordinates of the forearm connected to the hand and the upper arm.
At this stage, it is assumed that they are connected by a long axis having a predetermined length. The lengths of the long axes of the hand, upper arm, and forearm are extended by a predetermined length (for example, 10%), and the circle of the forearm (expanded by 10%) is moved so that these three circles are not separated.

The locus drawn by the central coordinates at this time is an area where a minute change can be given while maintaining the coupling. However, the movement of the circle that does not separate the target portion of the end point and the portion connected to it is performed so as not to exceed the movable range angle between the joints based on the model. By adjusting the short axis by the same ratio as the long axis, the proportion can be kept constant.

By carrying out the above-mentioned operation, since the application of the minute change is limited, the optimization processing can be speeded up,
It is possible to prevent separation of each part.

Further, according to the present invention, the motion of a moving body whose joints are connected by parts, such as an articulated robot or an animal, can be analyzed by the same method. For example, when analyzing the motion of an articulated robot, the following is performed. The knowledge of the shape of the robot (size / connection relationship), the knowledge of the motion of each part functionalized, and the possible existence area of each part are stored as the knowledge of the motion of the robot.

The silhouette image is extracted by calculating the difference between the background image and the input image. By performing the AND process of the silhouette image and the possible area, the rough parameters of each part are calculated. This parameter is corrected using the model, and the optimization process is performed. By this method, the movement of the robot can be analyzed and optimized based on this.

Further, optimization without using the silhouette image of the shape generator 5 is also conceivable.

As an example, the following judgment criteria can be mentioned. The difference between the parameter group of the model and the parameter group of each part of the target object in the time-series image is less than or equal to a threshold value, and the parameters of the same part in temporally previous and subsequent images and each target object in the time-series image If the difference between the parameter groups of the parts is less than or equal to a threshold value, the replacement is not performed.

The parameters calculated by the matching unit 8 are output to the image generation unit 12 under the control of the control unit 11. The image generation unit 12 generates, for example, an ellipse based on the parameters calculated by the matching unit 8 and images the parameters of each part. However, instead of an ellipse,
Imaging may be performed with a line segment or a cylinder. The output of the analysis result imaged by the image generation unit 12 is transferred to the D / A unit 9 and converted into an analog signal. Further, based on the signal from the control unit 11, the analog signal is output to the output unit 10 such as a display and the image analysis result is displayed on the display or the like. That is, the analysis result shown in FIG. 9B is obtained for the time series image shown in FIG. 9A.

The time-series image analysis device as described above can reduce the amount of information by formulating the motion parameters of the object, and can be applied to many subjects by calibrating the size and position of each part of the model. Yes, you can correct the difference in size due to the difference in shooting distance. In addition, if the shape or motion of an object cannot be accurately captured due to the effect of occlusion or shadow, correction is performed using the knowledge of the shape change due to the shape or motion of the object, and the motion of the object can be performed with high accuracy. Can be analyzed. Further, by utilizing the periodicity, the information amount of the model can be reduced, and by calibrating the period, it is possible to deal with the case where the speed is different in the same operation.

In the above description, the synchronization unit 21 determines the synchronization time and performs the motion matching. However, this processing is omitted, and all combinations of the time series image group and the model parameter group are searched. May be performed.

Further, although the period calibrating unit 22 calibrates the period for the operation having the periodicity, this process may be omitted. In addition, the period calibration unit 22 includes the synchronization unit 2
Not performed if 1 is omitted.

Although the parameter correction unit 23 corrects the parameters, this process can be omitted. Further, in the present embodiment, an ellipse is used as a simple geometrical shape expressing each part, but it may be described by a line segment or a cylinder.

Next, the time series image analysis device of the second embodiment will be described.

The configuration of the second embodiment is basically the same as that of the first embodiment, and only different parts will be described here.

FIG. 10 shows a concrete configuration example of the parameter calculation section 7 in the second embodiment. Here, the same components as those in the first embodiment are designated by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

The parameter calculation unit 7 includes a shape generation unit 5
The output from the target object model unit 6 is transferred to the candidate region calculation unit 31. The output of the candidate area calculation unit 31 is transferred to the matching unit 8.

In order to perform the optimization process at high speed, the parameters of the position and inclination angle of each part of the object are roughly calculated.

Candidate regions for each part of the object in the time-series image are calculated, the connection relation between the candidate regions is calculated from the shape of the calculated candidate region and the object, and each position or inclination of the region group having the connection relation is calculated. Calculation of the corner parameters and the possible range in which each parameter exists. Hereinafter, a possible range in which each parameter exists will be referred to as an allowable range.

FIG. 11 shows a concept in calculating a candidate area of each part of the object in the time-series image. Regarding the method of calculating the parameters, refer to the flowchart of FIG.
The candidate area calculation unit 31 will be described.

The parameter group of the possible area of each part stored in the object model section 6 is imaged (step S51). Specifically, an image is generated using parameters (center point, radius, rotation angle, etc.) such as circles and ellipses that are geometrical shapes stored in the object model unit 6. For example, 1 is set to the pixel (x, y) that satisfies the expression (3).
If is not satisfied, “0” is stored and a binary image is generated.

Next, the candidate area of each part is calculated using the possible area and the silhouette image imaged as shown in FIG. 3 (step S52). A rectangle circumscribing the silhouette image shown in FIG. 11B is obtained, and the center line of this rectangle is obtained. In addition, the position of the center line is calculated from the rectangle circumscribing the possible existence area imaged in step S52, and the AND output is obtained in a manner in which the center lines of the two coincide.

The result is a candidate area in which each site exists as shown in FIG. 11 (c). An end point region is selected from the extracted candidate regions of each part (step S53).
The end point area is an area (hand or foot) that is connected to another area in only one direction. First, the center of gravity (x, y) of the end point region is calculated (step S54). The method of calculating the center of gravity is, for example, a value obtained by adding the x and y coordinates of the end point area and dividing each added value by the number of pixels in the end point area.

FIG. 12 shows the concept of the processing for obtaining the connection relationship between the candidate areas and the shape of the target area and calculating the respective positions, inclination angles, and allowable ranges of the area groups having the connection relationship. The position (center point) of each part of the object of the time-series image is calculated as follows.

First, a connection area having a connection relationship with the end point area is calculated (step S55). Specifically, the connection relation is obtained from the size l of the part of the end point region, the size l ′ of the part of the other region connected to the end point region, and the shape of the object in the object model unit 6. If the object is a human, the part that has a connection with the hand, which is the end point region, is the forearm, so l,
The length of the long axis of each part is stored in l '.

Then, the connection relation of each area is calculated as follows. Radius R centered on the center of gravity of the end point area
A circle of (= (l + l ') / 2) is drawn, and a region existing on the circle is defined as a region that is joined with the end point region, that is, a joined region. The center of gravity of the portion where the circle and the combined area overlap is set as the center point (x, y) of the combined area (step S56).

FIG. 12 (b) is an area overlapping with a circle having a radius (l + l ') / 2 with the center of gravity of the end point area as the center point, and the thick line of the combined area in FIG. 12 (d) is the circle and the combined area. Is the overlapping part. Further, 1/2 of the length of the arc where the arc overlaps the joining area is set as the allowable range ε (step S5).
7). That is, since there is a possibility that the center point can be anywhere in the thick line (overlapping area) of the combined area in FIG. 12D, ½ of this overlapping portion is ε, and the coordinates of the center point are (x ± ε, y ± ε).

FIG. 12D shows a schematic diagram of a method of calculating the allowable range ε. It is determined whether or not the binding regions of all the sites have been detected (step S58), and when the binding regions of all the sites have not been detected (NO), the binding regions are set as new end point regions (step S59), and the result is fed back to step S55. .

When detection is performed for all parts (Y
ES), and the inclination angle corresponding to the center of gravity of each part is calculated (step S60). Specifically, an ellipse centered at the point (x, y) calculated above at 5 ° intervals from 0 ° to 180 ° is generated, and the angle at which the matching error with the silhouette image is minimized is the inclination angle. And

FIG. 12E shows the outline of the method of calculating the tilt angle. However, when multiple bond areas are detected when calculating the bond area, information such as the shape of the object and the inclination angle between the parts in the bond relationship is used, and if there is a contradiction, the bond Discard the relationship and narrow down the connection relationship.

Incidentally, even when the moving direction of the object in the time-series image and the moving direction of the model stored in the object model unit 6 are opposite, the following can be applied.

First, the moving direction of the object is detected from the time series images. The moving direction can be detected by, for example, calculating the center of gravity of the silhouette image and changing the x coordinate of the center of gravity. It is determined whether the moving direction of the target object matches the moving direction of the motion stored in the target object model unit 6.

In the case of the same direction, the parameters of the model are used as they are, and in the case of the opposite direction, the following processing is performed.
Reverse the behavior of the model.

A specific example of a method of reversing the motion of the model when the object is a human is shown below. FIG. 49
[Fig. 3] is a conceptual diagram regarding model inversion.

The position of the body part at each time (x2
, Y2) The coordinates are calculated. With respect to the part i other than the body part (i = 2), the difference between the body part and each parameter is calculated as Δx5 = (x2-x5) or Δy5 = (y2 in FIG. 49.
-Y5). Body parameters (x2, y
2 and θ 2) is added to the previously calculated difference, and the addition result is stored in the opposite part (for example, the right hand is the left hand) corresponding to the part. That is, x8 = Δx5 + x2, y8 = Δy5 + y
Do as in 2.

By doing in this way, it is also possible to analyze the motion of the object that is moving in the opposite direction in one model. In the above description, an ellipse is used as a simple geometrical shape that expresses each part, but this can be described not only by an ellipse but also by a line segment or a cylinder.

In FIG. 12 (e), an ellipse is used as a simple geometrical shape for expressing each part, but this can be described not only by an ellipse but also by a line segment or a cylinder. In the present embodiment, the load of the optimization process can be reduced and the processing time can be shortened by roughly calculating the parameters of each part of the object in the time-series image.

Next, a time series image analysis apparatus as the third embodiment will be described.

FIG. 25 shows an example of the overall configuration of a time series image analysis device as a third embodiment. The configuration of the present embodiment is basically the same as that of the first embodiment, the same components are designated by the same reference numerals, and only different portions will be described.

In this time-series image analysis device, the input signal from the input unit 1 is the A / D unit 2 and the buffer memory unit 3.
Is stored in the auxiliary input / output unit 4 via The signal of the auxiliary input / output unit 4 is output to the shape generation unit 5. Shape generator 5
Is transferred to the object model unit 6 and the matching unit 41. The object model unit 6 is connected to the matching unit 41. The parameters calculated by the matching unit 8 are output to the image generation unit 12 under the control of the control unit 11. The output of the analysis result imaged by the image generation unit 12 is output to the output unit 10 via the D / A unit 9. Control unit 1
The control signal 1 controls the input unit 1, the auxiliary input / output unit 4, the object model unit 6, the matching unit 41, and the output unit 10, respectively.

Such a time-series image analysis device is suitable for high-speed analysis processing, and does not calculate the approximate parameters of the position and tilt angle of each part of the object, but the model parameters are calculated for each object. It is treated as the initial parameter value of the part.

First, the input unit 1 such as a CCD camera, based on the control signal from the control unit 11, outputs the video signal of a predetermined time A
/ Transfer to D section 2. The A / D unit 2 converts the video signal from the input unit 1 into a digital signal and transfers the digital signal to the buffer memory unit 3. The buffer memory unit 3 is composed of a plurality of frame memories, stores the digital signal from the A / D unit 2, and sequentially outputs the digital signal to the auxiliary input / output unit 4.

The auxiliary input / output unit 4 outputs the digital signal to the shape generator 5 in response to the control signal from the controller 11. In the shape generation unit 5, after calculating the absolute value of the difference between the background image in which the object does not exist and the original image in which the object exists, the absolute value is binarized, and only the moving area is detected. Taken out. This moving area becomes a silhouette image of the object. Here, as the background image, a previously captured image may be used, or a background image generated from the original image may be used.

The silhouette image generated by the shape generating section 5 is transferred to the object model section 6 and the matching section 41.
The object model unit 6 stores information on the shape and motion of the object. In addition, by the control signal of the control unit 11,
Size calibration is performed as described in the previous embodiment.
The silhouette image generated by the shape generation unit 5 and the model of the object model unit 6 are output to the matching unit 41. Matching unit 4
In 1, the outputs of the shape generation unit 5 and the object model unit 6 synchronize the time regarding the operation of the object in the time-series image and the object model unit 6, and the entire parameter group of the object model unit 6 is shaped. Match with the silhouette image group of the generation unit 5.

Then, the parameters calculated by the matching section 41 are output to the image generation section 12 by the control signal from the control section 11. The output of the analysis result imaged by the image generation unit 12 is transferred to the D / A unit 9 and converted into an analog signal. Further, based on the control signal from the control unit 11, the analog signal is output to the output unit 10 such as a display,
Display image analysis results.

Next, FIG. 26 shows a block configuration in the processing of the matching section 41.

The outputs of the former shape generator 5 and the object model unit 6 are transferred to the synchronization unit 21 of the matching unit 41. The synchronization unit 21 synchronizes the time in the time-series image with respect to the operation of the target object and the target object model unit 6.

The outputs of the object model unit 6 and the synchronizing unit 21 are transferred to the period calibrating unit 22. In the cycle calibrating unit 22, when the target operation has a periodicity by the control signal from the control unit 11, the time required for one cycle is calculated based on the plurality of synchronization times calculated by the synchronizing unit 21 and the model. Calculate and calibrate the parameters of the model of the position and tilt angle of each part. Then, the outputs of the shape generation unit 5 and the object model unit 6 are transferred to the optimization processing unit 51. Optimization processing unit 51
Then, the entire parameter group of the object model unit 6 is matched with the silhouette image group of the shape generation unit 5.

The optimization processing section 51 matches the model parameter group of the object model section 6 with the silhouette image group generated by the shape generation section 5. As an example of a method for this, an evaluation function is set and minimized. For example,
In the case of an analysis targeting a human, the evaluation function E _{T is} defined as shown in Expression (7), and the evaluation function E _T is minimized by using the annealing, the steepest descent method, or the like as in the above-described embodiment. The optimum parameters of the parameter group of the series image are calculated.

[0130]

(Equation 7)

Here, T is the period, and i = T to T + 1 is the time. Sil is a silhouette image of the object in the time-series image, Par is an elliptical image generated from the model parameters, and the first term has the function of matching the model parameters with the silhouette image. Pc is time i-2, i-
1, i, i + 1, i + 2 time series image parameters,
Weights are given as 1: 2: 3: 2: 1, respectively, and are average values. The second term has the effect of matching the parameters of the time-series image with the parameter values estimated from the preceding and following times. k is a weight for each term.

In the above description, an ellipse is used as a simple geometrical shape expressing each part, but this can also be described by a line segment or a cylinder.

In the above embodiment, the structure is simplified and the initial parameters of the time series images are not calculated.
Processing can be performed faster than in the embodiment.

Next, a time series image analysis apparatus as the fourth embodiment will be described.

The overall configuration of this embodiment is the same as that of the third embodiment, and only the matching section 41 shown in FIG. 27, which has a different configuration, will be described.

In the matching section 41, the shape generation section 5
The output of the object model unit 6 is transferred to the optimization processing unit 61. The optimization processing unit 61 uses a neural network to calculate a parameter group (position) of each part of the time-series image that is optimum for the silhouette image of the time-series image. The outputs of the object model unit 6 and the optimization processing unit 61 are transferred to the inclination angle calculation unit 62, and the rotation angle of each part of the time series image that is optimum for the silhouette image of the time series image is calculated. The output of the tilt angle calculation unit 62 is output to the image generation unit 12.

As shown in FIG. 27, the silhouette image of the time-series image of the shape generation unit 5, the model parameter group of the object model unit 6 and the output of the silhouette image group are transferred to the optimization processing unit 61. The optimization processing unit 61 calculates the parameter group of the position of each part of the time-series image that is optimal for the silhouette image of the time-series image using a hierarchical neural network. The parameter group (position) of each site calculated by the optimization processing unit 61 is transferred to the tilt angle calculation unit 62, and the tilt angle of each site of the time series image that is most suitable for the silhouette image of the time series image is calculated. . The parameter group calculated by the tilt angle calculation unit 62 is output to the image generation unit 12.

FIG. 28 (a) is a conceptual diagram regarding a learning method of a three-layer hierarchical neural network. The silhouette image of the model stored in the object model unit 6
Input to the input layer. The weighting of each unit is determined by learning so that the result via the network becomes a parameter of the position of each part of the model. The learning is performed on the image for one cycle of the operation, and the weight is updated. Note that the learning method can be performed by using, for example, a known back propagation method. By using this weight, the position of each part of the captured image is calculated. By inputting the silhouette images of the time-series images of the shape generation unit 5 into the input layer and passing them through the network, it is possible to calculate the position of each part that is optimum for the silhouette image. FIG. 28B is a conceptual diagram regarding parameter calculation for a captured image.

In the inclination angle calculation unit 62, the optimization processing unit 61
Of the parameter group (position) of each part calculated in step S, the size of each part stored in the object model unit 6 (short axis of ellipse, length of long axis) and time series image of the shape generation unit 5. Using the silhouette group, the tilt angle of each part of the time series image group is calculated. The position (x, y) of each part calculated by the optimization processing unit 61 is the center point, and the size of each part of the object model unit 6 (short axis b / 2 of the ellipse, length a / 2 of the long axis). ) Is used to generate an ellipse. For the tilt angle, generate an ellipse using the above parameters at 5 ° intervals from 0 ° to 180 °,
The angle at which the matching error with the silhouette image is minimized is the tilt angle. In addition, the calculation of the tilt angle is performed by the above-described FIG.
The concept is shown in (d).

Then, an elliptic image is generated from the calculated parameter group of each part and is transferred to the image generating section 9.

In the fourth embodiment, by using the neural network, the configuration of the optimization processing unit 24 is simplified, and the learning is obtained by learning even if the optimization parameters are not adjusted intentionally. be able to.

Next, a time series image analysis apparatus as the fifth embodiment will be described.

FIG. 29 shows an example of the overall configuration of the time-series image analysis device. The overall configuration of this embodiment is the same as that of the third embodiment, and the same components are designated by the same reference numerals and described.

In this time-series image analysis device, the input signal from the input section 1 is stored in the auxiliary input / output section 4 via the A / D section 2 and the buffer memory section 3. The signal of the auxiliary input / output unit 4 is output to the shape generation unit 5. The output of the shape generation unit 5 is transferred to the object model unit 71, the parameter calculation unit 7, and the matching unit 72. The target object model section 71
Are connected to the parameter calculation unit 7 and the matching unit 72.

The output of the parameter calculation section 7 is connected to the matching section 72. The output of the matching unit 72 is output to the object model unit 71 and the image generation unit 12. The output of the image generation unit 12 is output to the output unit 10 via the D / A unit 9. The control signal from the control unit 11 controls the input unit 1, the auxiliary input / output unit 4, the object model unit 71, the parameter calculation unit 7, the matching unit 72, and the output unit 10, respectively. The time-series image analysis device adaptively learns the model as the processing progresses.

In this time series image analysis device, the CCD
The input unit 1 such as a camera transfers a video signal for a predetermined time to the A / D unit 2 under the control of the control unit 11. The A / D unit 2 converts the video signal from the input unit 1 into a digital signal and transfers it to the buffer memory unit 3.

The buffer memory unit 3 is composed of a plurality of frame memories, stores the digital signals from the A / D unit 2, and sequentially outputs the digital signals to the auxiliary input / output unit 4 such as a magnetic disk device. The auxiliary input / output unit 4 is the control unit 1.
The digital signal is output to the shape generation unit 5 according to the control signal from 1. The shape generator 5 extracts the silhouette area of the object.

Next, the silhouette image generated by the shape generation unit 5 is transferred to the object model unit 71, the parameter calculation unit 7, and the matching unit 72. The object model section 71 stores information on the shape and motion of the object. Further, the information regarding the operation is updated based on the output of the matching unit 72.
Under the control of the control unit 11, the parameter calculation unit 7 roughly determines the parameters of the position and the inclination angle of each part of the object from the silhouette image of the shape generation unit 5 and the output of the object model unit 71.

Then, the silhouette image generated by the shape generation unit 5, the model of the object model unit 71, and the parameters of each part calculated by the parameter calculation unit 7 are stored in the matching unit 72.
Is output to In the matching unit 72, the outputs of the shape generation unit 5, the object model unit 71, and the parameter calculation unit 7 synchronize the times of the object in the time-series image and the model of the object model unit 71, and the parameter calculation unit 7 The entire parameter group calculated by is matched with the entire parameter group of the target object model unit 71 and the silhouette image group of the shape generation unit 5. The output of the matching unit 72 is transferred to the object model unit 71 and the image generation unit 12 under the control of the control unit 11.

The output of the image generator 12 is the D / A unit 9
To convert to an analog signal. Furthermore, the control unit 11
The analog signal is output to the output unit 10 such as a display based on the signal from the image display unit to display the image analysis result.

The object model section 71 and the matching section 72 will be described below.

FIG. 30 shows the concept of updating model parameters, and FIG. 31 shows the configuration of the object model section 71.

First, under the control of the control unit 11, the outputs of the shape knowledge unit 81 and the motion knowledge unit 82 are output to the parameter calculation unit 7.
Transfer to. The rough parameters calculated by the parameter calculation unit 7 and the outputs of the shape knowledge unit 81 and the motion knowledge unit 82 are output to the matching unit 72. The output of the matching unit 72 is fed back to the operation knowledge unit 82 based on the control signal from the control unit 11.

In the object model section 71, in the first one cycle of the time series image, the parameter calculation section 7 uses the initial model of the shape knowledge section 81 and the motion knowledge section 82 given from the outside as in the first embodiment. , To the matching unit 72.

After the second cycle, the correction parameters calculated by the matching unit 72 are stored, and the model given from the outside is separately stored as an initial model. The model of the motion knowledge unit 82 is updated based on this correction parameter,
In the subsequent processing, the updated model is output to each processing unit. The model updating method is, for example, a value obtained by adding a correction parameter to each parameter of each part in the model before updating.

FIG. 32 is a block diagram of the processing of the matching section 72. The outputs of the shape generation unit 5 and the object model unit 71 are transferred to the synchronization unit 21. The output of the synchronization unit 21 is
It is transferred to the period calibration unit 22. The shape generation unit 5, the object model unit 71, the parameter calculation unit 7, the synchronization unit 21, and the period calibration unit 22 are output to the parameter correction unit 23. The outputs of the shape generation unit 5, the object model unit 71, and the parameter correction unit 23 are transferred to the optimization processing unit 24. Optimization processing unit 2
The output of No. 4 is transferred to the image generation unit 12, and the outputs of the optimization processing unit 24 and the object model unit 71 are transferred to the correction parameter calculation unit 91. The output of the correction parameter calculation unit 91 is fed back to the object model unit 71.

In the synchronizing section 21, the object model section 71
Based on the output of the shape generation unit 5, the times of the object in the time-series image and the model of the object model unit 71 are synchronized. The cycle calibrating unit 22 calibrates the parameters of the model of the position and the tilt angle of each part when the target operation has periodicity. The parameter correction unit 23 corrects the parameters of the time-series image by comparing with the model in one image.

Then, the optimization processing section 24 calculates optimum parameters for the object in the time series images. The output of the optimization processing unit 24 is controlled by the control signal from the control unit 11.
It is output to the image generation unit 12 and the correction parameter calculation unit 91. In the correction parameter calculation unit 91, the optimization processing unit 24
Difference information of each parameter of each part between the parameter calculated in step 1 and the model parameter of the object model part 71 is obtained. This difference information is fed back to the object model unit 71 as a model correction parameter.

When the optimum parameters for the object are calculated using the same method as in the above-described embodiment, the difference information of each parameter is stored as a vector so that the direction of this vector is weighted and minute You can make a difference. Further, after the third cycle, it is only necessary to replace the portion of the correction parameter table.

FIG. 30B is a conceptual diagram relating to how to make a minute change in the optimization process.

In the fifth embodiment, by updating the model as needed, a model suitable for the object to be analyzed can be created, and the processing can be performed with high accuracy and at high speed.

Next, a time series image analysis apparatus as a sixth embodiment will be described.

The overall configuration of the present embodiment is the same as that of the fifth embodiment, and an object model part having a different configuration will be described. The present embodiment relates to a model in which a model of each part is hierarchically divided and can be exchanged for each part.

FIG. 33 shows a concrete example of the configuration of the object model section of this embodiment.

In this object model part, the outputs of the shape knowledge part 81 and the motion knowledge part 101 are transferred to the parameter calculation part 7, and the outputs of the shape knowledge part 81, the motion knowledge part 101 and the parameter calculation part 7 are transferred to the matching part 72. To be done. The output of the matching unit 72 is output from the operation knowledge unit 10 according to a control signal from the control unit 11.
It is output to 1.

The object model section outputs the output of the shape knowledge section 81 and the motion knowledge section 101 hierarchically stored for each part to the parameter calculation section 7 and the matching section 72 according to the control signal of the control section 11. Forward. The rough parameters calculated by the parameter calculation unit 7 and the outputs of the shape knowledge unit 81 and the motion knowledge unit 101 are output to the matching unit 72. Matching unit 72
Then, the part that matches the model and the part that switches the model are extracted. Based on the control of the control unit 11, the matching unit 7
The output of 2 is fed back to the motion knowledge unit 101.

Then, the object model section 71 hierarchically stores the connection with other parts regarding the center part of the shape structure of the object. FIG. 34 shows a tree structure in which other parts are connected based on the body part of a human being as an example. In this way, the parameters of the first stored model are extracted from the hierarchical model and output to the parameter calculation unit 7 to roughly calculate the parameters. The calculated rough parameters and model (shape knowledge unit 81 and motion knowledge unit 101) are output to the matching unit 72 to detect the part to be switched. The next model stored under the hierarchical structure of the switching portion detected by the matching unit 72 is output to the parameter calculation unit 7 and the matching unit 72.

In the matching section 72, the object model section 71
The optimum solution of the parameters of the time series image is calculated from the model output from. In the optimization process, in order to judge the suitability, in the above embodiment, the sum of all 14 types of sites is taken in the evaluation function E as shown in equation (6) (j = 1.
~ 14). This is decomposed into the formula for each part, and 1
Obtain four types of evaluation functions. An evaluation standard is set for each evaluation function, and a site where the evaluation function E is larger than the evaluation standard is selected.

Specifically, a threshold serving as an evaluation standard is set, and it is determined whether the evaluation function E of each part exceeds a predetermined threshold. The evaluation in this case is performed from the upper part of the tree structure. In other words, in the case of human beings, it starts from the body.
The evaluation function E exceeds the threshold and outputs the selected part to the motion knowledge unit 101. This operation is performed until the evaluation function of each part does not exceed the threshold value. In addition, after the model is determined, when the evaluation function of each part becomes larger than the threshold value, the model is switched by the above method.

According to the sixth embodiment, the information amount of the model can be reduced, and more motions can be analyzed.

Next, a time series image analysis apparatus as the seventh embodiment will be described.

The overall configuration of this embodiment is the same as that of the first embodiment, and a different configuration will be described. The present embodiment relates to a method of selecting an optimal model for an analysis target image from a plurality of models.

FIG. 35 shows the structure of the time-series image analysis device according to the seventh embodiment, and the same components will be assigned the same reference numerals and described.

In this time series image analysis device, the input signal from the input unit 1 is the A / D unit 2 and the buffer memory unit 3.
Is stored in the auxiliary input / output unit 4 via the, and the output signal of the auxiliary input / output unit 4 is output to the shape generation unit 5. The output of the shape generation unit 5 is transferred to the object model unit 6 and the analysis unit 111.

The object model unit 6 is connected to the analysis unit 111. The output of the analysis unit 111 is connected to the model determination unit 112 and the image generation unit 12. Model determination unit 112
Is output to the object model unit 6. The output of the image generation unit 12 is output to the output unit 10 via the D / A unit 9. Further, according to a control signal from the control unit 11, the input unit 1, the auxiliary input / output unit 4, the object model unit 6, the analysis unit 1
11, the model determination unit 112, and the output unit 10 are controlled.

In such a time series image analysis device,
The input unit 1 such as a CCD camera transfers a video signal for a predetermined time to the A / D unit 2 under the control of the control unit 11. A
The / D section 2 converts the video signal from the input section 1 into a digital signal and transfers it to the buffer memory section 3. The buffer memory unit 3 is composed of a plurality of frame memories, and the A / D
The digital signal from the section 2 is stored and sequentially output to the auxiliary input / output section 4.

The auxiliary input / output section 4 outputs the digital signal to the shape generation section 5 in response to a control signal from the control section 11. The shape generator 5 extracts the silhouette area of the object. The silhouette image generated by the shape generation unit 5 is transferred to the object model unit 6 and the analysis unit 111. The object model unit 6 stores information such as the shape of the object and a plurality of motions.

Further, the analysis unit 111 analyzes a plurality of models and calculates the parameters of the position and inclination angle of each part of the object. The parameters of each part for the plurality of models calculated by the analysis unit 111 are output to the model determination unit 112 and the image generation unit 12 based on the control signal from the control unit 11. The model determination unit 112 extracts the model having the minimum value of the evaluation function calculated for the plurality of models based on the output of the analysis unit 111, and determines the model.

Then, the model determined by the model determination unit 112 is output to the object model unit 6, and the information about the determined model is output from the object model unit 6 for the subsequent analysis. Under the control of the control unit 11, the output of the analysis unit 111 is transferred to the D / A unit 9 via the image generation unit 12 and converted into an analog signal. Further control unit 1
The analog signal is output to the output unit 10 such as a display according to the control signal from No. 1 to display the image analysis result.

FIG. 36 shows a block configuration in the processing of the above-mentioned analysis unit 111, and FIG. 37 shows a concept regarding a method of selecting an optimum model from a plurality of models for an object in a time series image.

The silhouette image of the shape generating section 5 is output to the direction determining section 121, and the output of the direction determining section 121 is fed back to the object model section 6. This shape generator 5
Of the silhouette image and each model of the object model section 6,
It is output to the parameter calculation unit 122. The output of the parameter calculation unit 122 is transferred to the matching unit 123, and the control unit 1
It is output to the model determination unit 112 and the image generation unit 12 according to the control signal from 1.

Then, the direction determining unit 121 detects the moving direction of the object based on the area of the silhouette image of the shape generating unit 5. Specifically, when the area of the silhouette image does not change with time, it moves in a direction perpendicular to the camera optical axis direction, and when it increases or decreases with time, it moves in parallel or diagonally with respect to the camera optical axis direction. There is.

That is, the moving direction is determined by the rate of increase / decrease in the area of the silhouette image. The detected moving direction is fed back to the object model unit 6. The moving direction of the object can be detected by detecting the center of gravity of the silhouette as a vector, or the area or center of the circumscribed quadrangle that includes the silhouette image can be detected. A plurality of models corresponding to the silhouette image of the shape generation unit 5 and the detected moving direction of the object model unit 6 are output to the parameter calculation unit 122.

For example, in the case of motion analysis of a person, a plurality of models such as walking, running, sitting, and still are stored in the object model unit 6, and a model relating to the shape and the direction determining unit 12 are stored.
The model related to the motion in the direction detected in 1 is output to the parameter calculation unit 122. The parameter calculation unit 122 roughly calculates the parameters by using each model as in the parameter calculation unit 7 in the above-described embodiment. The matching unit 123 calculates the optimum parameters for the target object in the time-series image from the output of the parameter calculation unit 122 and each model of the shape generation unit 5 and the target object model unit 6, as in the matching unit 8 of the embodiment. Evaluation function E after optimization processing
(Matching error with model) is calculated. that time,
If it is not synchronized with the operation, that is, if the area error in the synchronization unit 21 in the above embodiment is larger than a predetermined threshold value, the model is discarded and the subsequent processing is not performed.

Then, the optimization process is performed on the synchronized model in the same manner as in the above embodiment. The output of the matching unit 123 is transferred to the model determination unit 112 and the image generation unit 12 according to a control signal from the control unit 11. However, the number of parameter calculation units and matching units can be increased by the number of models stored in the object model unit 6, and processing can be performed in parallel. If the model has been decided,
The output of the matching unit 123 is not transferred to the model determination unit 112, and only one parameter calculation unit and one matching unit are used.

In the model determining unit 112, the matching unit 12
The smallest one is selected from the plurality of evaluation functions E output from 3 as the optimum model. The selected model is fed back to the target object model unit 6, and the model determined by the target object model unit 6 is output for the subsequent analysis. After that, the process is continued using the selected model, and when the evaluation function E exceeds a predetermined value, the model is reselected again. In addition, by such processing,
Analysis can be performed even when the motion of the object is not determined. This process is not parallel, and it is also possible to calculate the error of the optimization process for each model and finally select the model with the smallest error.

Next, a time series image analysis apparatus as an eighth embodiment will be described.

The overall configuration of this embodiment is the same as that of the second embodiment, and a different configuration will be described. In the present embodiment, when the parameters of a certain part are determined by considering the priority order of processing, the candidate regions of all the parts having a connection relation with the part are further limited, and the optimization process is performed at high speed. .

FIG. 38 shows the configuration of the parameter calculation unit in the time-series image analysis device of the eighth embodiment, and the same components will be assigned the same reference numerals and described.

In this parameter calculation unit, the outputs of the object model unit 6 and the shape generation unit 5 are the candidate region calculation unit 1
31 is transferred. The candidate area calculation unit 131 calculates a candidate area of each part of the target object in the time-series image, preferentially processes a part having a small error, that is, a small area of the candidate area, and selects a candidate area of another part. Make a limitation. The output of the candidate area calculation unit 131 is transferred to the candidate area calculation unit 131 and the matching unit 8.

FIG. 39 shows the introduction of processing priority.
It is a figure which shows the concept in calculation of the presence candidate area | region of each part of the target object in a time series image.

First, the candidate area calculation unit 131 obtains the candidate area 1 by AND processing of the possible existence area 1 and the silhouette image similarly to the candidate area calculation unit 31 of the second embodiment. A region in the candidate region 1 that can be estimated to have a small error,
In other words, for a candidate area smaller than a predetermined threshold of area, for example, in the case of a human, an ellipse centered on the center of gravity of the candidate area is generated for a head, a body, and a hand, and matching with a silhouette image is performed. Calculate parameters.

When the positions of these three points are determined, the regions where other parts can exist are limited. In other words, a more limited candidate area is calculated by using constraint conditions such as the joint angle between the determined part and the part having a connection relationship and the length of each part.

Specifically, when the object is a human, the head / body / hand part having a small area of the candidate area 1 is preferentially processed to determine the position. When the positions of the head and the body are determined, the possible area 1 becomes a limited area such as the possible area 2 based on the length of each part. That is, the position of the shoulder is calculated from the center point of the body part using the equation (2). Then, consider a circle having a radius r centered on the position of the shoulder. Here, the radius r refers to the area of the upper arm (length of the upper arm /
2) Regarding the area of the forearm (upper arm + forearm /
2) Regarding the hand, it means (upper arm + forearm + hand / 2).

Next, the common part of the circle having the center point and the radius described above and the possible area 1 is calculated, and the possible area 2 is calculated.
Is extracted. Like the second embodiment, the possible area 2
The candidate area 2 is calculated by AND processing of the silhouette image, and the position, inclination angle, and allowable range of each part are calculated using this area.

A method of directly extracting the candidate area 2 from the candidate area 1 as shown by the broken line in FIG. 39 is shown below.

First, the position of the shoulder from the determined center point of the body portion is calculated by the equation (2). For example, when the forearm region is limited by using the length between the parts, a radius (the length of the hand / 2 the length of the forearm / the center of the center point of the hand or shoulder)
2 + ε0), (upper arm length + forearm length / 2 + ε0
) Consider the circle. Here, for example, 10% of the area of the candidate region of the hand portion is set as ε0. The common area between the two circles and the candidate area 1 is referred to as a candidate area 2.

Similarly, for other regions, the candidate region 1
The more limited candidate area 2 is calculated. The angle formed by the shoulder and the hand is larger than the angle formed by the shoulder and the forearm and between the shoulder and the upper arm. Further, the angle between the shoulder and the forearm is larger than the angle between the shoulder and the upper arm, and a more limited candidate region 2 can be obtained from the constraint condition of the angle between the joints in the above example.

FIG. 40 is a diagram showing a concept related to calculation of the limited candidate area 2. Second using candidate area 2
Similar to the embodiment, each position of each part, the tilt angle, and the allowable range are calculated.

In response to a control signal from the control unit 11, these parameter groups are output to the matching unit 8 to perform optimization processing. In the optimization process on the time axis, the confirmed head, body, and hand can be excluded from the processing target. Further, the processing can be speeded up by reducing the frequency of giving a minute change.

According to the eighth embodiment, the position of a highly accurate part is preferentially determined, and other parts are optimized from proper initial values based on this, so that the optimization process is performed. Can be speeded up.

Next, a time series image analysis apparatus as the ninth embodiment will be described.

FIG. 41 shows a specific structural example of the matching section 72 of the ninth embodiment. The overall configuration of this embodiment is
This is equivalent to the fifth embodiment, and a different configuration will be described. In the present embodiment, prediction is performed using past optimization information and high-speed processing is performed.

In the matching section 72, the outputs of the shape generating section 5 and the object model section 71 are input to the synchronizing section 21, and the outputs of the synchronizing section 21 and the object model section 71 are transferred to the period calibrating section 22. It Shape generation unit 5 and object model unit 7
1, parameter calculation unit 7, synchronization unit 21, and period calibration unit 22
The respective outputs of the above are sent to the parameter correction unit 23. The outputs of the shape generation unit 5, the object model unit 71, and the parameter correction unit 23 are transferred to the optimization processing unit 24. The output of the optimization processing unit 24 is the motion vector calculation unit 1
41 and the image generation unit 12. The output from the motion vector calculation unit 141 is fed back to the optimization processing unit 24.

Then, the synchronizing section 21 synchronizes the time of the operation in the object model section 71 with the object in the time series image based on the outputs of the object model section 71 and the shape generating section 5. When the target operation has a periodicity, the cycle calibration unit 22 calibrates the parameters of the model of the position and inclination angle of each part. The parameter correction unit 23 corrects the parameters of the time-series image by comparing with the model in one image.

FIG. 42 shows the concept of motion vector calculation, and FIG. 43 shows the concept of optimization processing using prediction.

In the optimization processing section 24, only the first few sheets (for example, five sheets) are subjected to the optimization processing by the same method as in the above embodiment. Based on the control of the control unit 11, the optimized parameter is output to the motion vector calculation unit 141.
The motion vector calculation unit 141 stores the parameters calculated by the optimization processing unit 24 for the past several sheets, and predicts the optimum solution (parameter) at the next time from them. As the prediction method, linear prediction may be used from the change of each parameter with respect to time. Further, it can be performed by obtaining a motion vector from the position / tilt angle of the part and using it. The parameter of the calculated next time is used by the optimization processing unit 2
4, the parameters of the model of the embodiment described above are replaced with the parameters predicted by the motion vector calculation unit 141, and optimization processing such as annealing is performed.

In the subsequent processing, the parameters predicted by the motion vector calculation unit 7 are used, and the parameter correction unit 23 and the parameter calculation unit 7 can be omitted. However, in this case, the outputs of the synchronization unit 21 and the synchronization calibration unit 22 are transferred to the optimization processing unit 24. Further, when the evaluation function E calculated by the optimization processing unit 24 exceeds a predetermined threshold value, the processing of the parameter calculation unit 7 and the parameter correction unit 23 is performed.

In the ninth embodiment, since the parameter predicted from the parameter at the previous time is used, it is possible to perform the optimization processing which has good temporal continuity.

Next, a time series image analysis device as a tenth embodiment will be described.

FIG. 44 shows a specific structural example of the matching section of the tenth embodiment. The overall configuration of this embodiment is equivalent to that of the fifth embodiment, and a different configuration will be described.

In this embodiment, the error between the parameter of the time series image of each part and the parameter of the model is calculated, and the way of making a minute change is changed according to the error, that is, the adjustment degree of the parameter.

In the matching section 72, the outputs of the shape generating section 5 and the object model section 71 are transferred to the synchronizing section 21. The outputs of the synchronization unit 21 and the object model unit 71 are transferred to the period calibration unit 22. The shape generation unit 5, the object model unit 71, the parameter calculation unit 7, the synchronization unit 21, and the period calibration unit 2
2 is output to the parameter correction unit 23. Shape generator 5
The outputs of the object model unit 71 and the parameter correction unit 23 are transferred to the optimization processing unit 24. The outputs of the optimization processing unit 24 and the object model unit 71 are output to the adjustment degree calculating unit 151 under the control of the control unit 11, and the outputs of the optimization processing unit 24 are
It is transferred to the image generation unit 12. The output from the adjustment degree calculation unit 151 is fed back to the object model unit 71.

Then, in the synchronizing section 21, the object model section 7
Based on 1 and the output of the shape generation unit 5, the time in the operation of the object model unit 71 and the object in the time-series image are synchronized. The cycle calibrating unit 22 calibrates the parameters of the model of the position and the tilt angle of each part when the target operation has periodicity. The parameter correction unit 23 corrects the parameters of the time-series image by comparing with the model in one image.

FIG. 45 is a diagram showing the concept of the adjustment degree calculation. That is, in the optimization processing unit 24, the parameter correction unit 2
Based on the output of No. 3, optimization processing such as arening is performed only in the first cycle by the same method as in the above embodiment. Control unit 1
The parameter calculated by the optimization processing unit 24 is output to the adjustment degree calculation unit 151 in response to the control signal from 1. The adjustment degree calculation unit 151 calculates the adjustment degree according to the difference between the parameter calculated by the optimization processing unit 24 and the parameter of the object model unit 71. Here, the adjustment degree indicates how the optimization processing unit 24 gives a minute change. As the method of calculating the adjustment degree, the difference between the parameter calculated by the optimization processing unit 24 and the parameter of the object model unit 71 is calculated, and 10% of the difference is used as the adjustment degree. That is, as the value of the degree of adjustment is larger, the parameter needs to be adjusted, so that the method of giving a minute change is set to be larger than that of other parts.

The calculated adjustment degree is output to the optimization processing section 24. The optimization processing unit 24 calculates a minute change based on the adjustment degree as follows. For example, a value obtained by multiplying the initial minute change given by using a random number by the adjustment degree to the minute change is set as a new minute change.

Next, in the second and subsequent cycles, optimization processing is performed using this adjustment degree. By the control signal from the control unit 11,
The parameters calculated by the optimization processing unit 24 are transferred to the image generation unit 12. It should be noted that the degree of adjustment may be given not for each part but for each image.
FIG. 45B is a conceptual diagram related to calculation of the adjustment degree of walking motion. For example, in the case of a walking motion, the degree of adjustment of each part is low and the degree of adjustment of the entire image is low when the hands and feet do not overlap each other. On the other hand, when both hands and feet overlap and the back hands and feet are hidden, the degree of adjustment is high. The degree of adjustment in image units can be the difference from the silhouette image by generating an elliptical image from the parameters obtained at the level of a still image. Further, when processing is performed on a cycle-by-cycle basis, it is possible to suppress the addition of minute changes to an image with a high degree of adjustment.

According to the tenth embodiment, the degree of adjustment of the parameters of each part, that is, the way of giving a minute change handled in the optimization processing is changed according to the error, so that the processing can be performed at high speed. .

Next, a time series image analysis apparatus as the eleventh embodiment will be described.

FIG. 46 shows a specific structural example of the matching section of the eleventh embodiment. The overall configuration of this embodiment is equivalent to that of the fifth embodiment, and a different configuration will be described.

In this embodiment, the correction information in the parameter correction unit 23 is used to limit how to give a minute change in the optimization process.

In the matching section 72, the outputs of the shape generating section 5 and the object model section 71 are transferred to the synchronizing section 21. The outputs of the synchronization unit 21 and the object model unit 7 are transferred to the period calibration unit 22. Shape generation unit 5 and object model unit 7
1, parameter calculation unit 7, synchronization unit 21, and period calibration unit 22
Is output to the parameter correction unit 23. The outputs of the shape generation unit 5, the object model unit 71, and the parameter correction unit 23 are
It is transferred to the optimization processing unit 24. The outputs of the parameter correction unit 23 and the optimization processing unit 24 are transferred to the certainty factor calculation unit 161, under control of the control unit 11, and the outputs of the optimization processing unit 24 are transferred to the image generation unit 12. The output from the certainty factor calculation unit 161 is fed back to the optimization processing unit 24.

Then, in the synchronizing section 21, the object model section 7
Based on 1 and the output of the shape generation unit 5, the time in the operation of the object model unit 71 and the object in the time-series image are synchronized. The cycle calibrating unit 22 calibrates the parameters of the model of the position and the tilt angle of each part when the target operation has periodicity. The parameter correction unit 23 corrects the parameters of the time-series image by comparing with the model in one image. A part (reliability: high) in which the difference between the parameters of the time-series image and the parameter of the model performed by the parameter correction unit 23 is determined to be less than or equal to a predetermined threshold (reliability: high), and a part in which the difference is greater than or equal to the threshold (reliable Sex: medium), and the three-level information of a portion (reliability: low) in which the difference is equal to or larger than the threshold value and the replacement with the model is discarded is transferred to the certainty factor calculation unit 161. FIG. 47
FIG. 6 is a diagram showing a concept regarding information obtained by the parameter correction unit 23.

First, the certainty factor calculation unit 161 changes how to make a minute change based on the three classifications. For example, in a portion where the difference is less than or equal to the threshold value, a small change in a circle having a radius r as the threshold value ε is given, and in the portion where the difference is greater than or equal to the threshold value and the replacement with the model is performed, there is a direction before the replacement (angle θ). In a circle having a radius r with a threshold ε other than, the region where the replacement with the model is abandoned and the difference is greater than the threshold gives a minute change in the circle with a radius r that is twice the threshold ε.

Next, the calculated radius r and angle θ are output to the optimization processing section 24. The optimization processing unit 24 determines whether or not a given minute change exists within a circle having a radius r and an angle θ, and if there is, a minute change is used to perform optimization. FIG. 48 is a conceptual diagram of how to make a minute change using the certainty factor. The output of the optimization processing unit 24 is transferred to the image generation unit 12 under the control of the control unit 11.

According to the eleventh embodiment, it is possible to speed up the processing by using the information regarding the correction of the parameters and limiting the range in which the minute change handled in the optimization processing is given.

Although described based on the above embodiments, the present invention also includes the following inventions.

(1) In a time-series image analysis device for analyzing a predetermined object in a time-series image, a silhouette image which is an object is extracted from a plurality of images within a predetermined time range in the time-series image. Shape generation means, object model means for storing knowledge about the shape and movement of the object due to movement as parameters for each of the parts constituting the object, and the object extracted by the shape generation means. From the parameter calculation means for calculating the parameter corresponding to the parameter stored in the object model means, the entire corresponding parameter calculated from the parameter calculation means and the entire parameter of the object model means are collated, and the error is A time-series image analysis device, comprising: matching means for matching by minimizing.

The embodiments relating to the present invention are as follows:
The seven embodiments correspond.

Therefore, the parameters of the parts constituting the predetermined object in the time-series image, which cannot be calculated due to the influence of the occlusion and the like, are matched with the parameters of the model to recognize the object, The movement of can be analyzed with high accuracy.

(2) In the object model means of the time-series image analysis device described in the item (1), the knowledge about the shape of the object is divided into a plurality of parts constituting the object, and each divided object is divided. A time-series image analysis device characterized in that the size of a part is defined as a ratio to the overall size of an object, and the connection relationship between parts is stored in a table. The embodiment relating to the present invention corresponds to the first embodiment.

(3) In the object model means of the time-series image analysis device described in any one of the above items (1) and (2), a moving region is extracted from the above-mentioned time-series image and the moving region is extracted. The rectangle circumscribing the area is calculated, and the size of each part is corrected based on the size of the rectangle in the vertical direction and the ratio of the total size of each part that is stored in advance, and the size of the object is calibrated. A time-series image analysis device characterized by:

The embodiment relating to the present invention corresponds to the first embodiment.

Therefore, by calibrating the size of the model, it can be applied to many subjects, and the difference in size due to the difference in shooting distance can be corrected.

(4) In the time-series image analysis device described in the item (1), the parameter may be the position, inclination, and size of an ellipse corresponding to each part of the object in the time series. Characteristic time-series image analysis device.

The embodiments relating to the present invention are:
The embodiments of 3, 4, 5, 6, 7, 8, 9, 10, 11 correspond.

Therefore, by expressing the shape of the object with an ellipse, it is easy to match the object in the time-series image.

(5) In the time series image analysis device described in any one of the above items (1) to (4), at least one of the parameters is a representative value of the parameter and its allowable range. A time-series image analysis device.

The embodiments relating to the present invention correspond to the second and eighth embodiments.

Therefore, when the above parameters are determined, the allowable range of the parameters of each part is calculated,
Flexibility of each calculated parameter solution can be provided.

(6) In the object model means of the time-series image analysis device described in the item (1), the knowledge about the shape change due to the movement of the object is the parameter of each part of the moving object. The time-series image analysis device is characterized in that the calculation is performed based on the movement of the marker attached to the part.

The embodiment relating to the present invention corresponds to the first embodiment.

(7) In the object model means of the time-series image analysis device described in the item (6), the marker is attached to the central position of each part of the object, the time-series image analysis. apparatus.

The embodiment relating to the present invention corresponds to the first embodiment.

(8) In the object model means of the time-series image analysis device according to the item (6), the marker is attached to a connection point between the respective parts of the object, the time-series image. Analyzer.

The embodiment relating to the present invention corresponds to the first embodiment.

(9) In the object model means of the time-series image analysis device described in any one of the above items (1) to (8), the operation parameter related to the shape change of the object is set as a time. A time-series image analysis device characterized by allocating on a shaft at predetermined time intervals.

The embodiments relating to the present invention are as follows:
The embodiments of 4, 5, 6, 7, 9, 10 correspond.

Therefore, by assigning the target operation parameter to the predetermined time interval on the time axis, it is possible to deal with the change in the operation speed of the object.

(10) In the object model means of the time series image analysis device according to any one of the above items (1) to (9), the above parameters are held as a table. Image analysis device.

The embodiments relating to the present invention are:
The embodiments of 4, 5, 6, 7, 9, 10, 11 correspond.

Therefore, the processing becomes easy by storing the parameters of the target operation as a table.

(11) In the object model means of the time-series image analysis device described in any one of the above items (1) to (9), it is possible to hold the above parameters as mathematical expressions by using an interpolation function. Characteristic time-series image analysis device.

The embodiments relating to the present invention are:
The embodiments of 4, 5, 6, 7, 9, 10, 11 correspond.

Therefore, the parameters of the target operation are
The data amount of the model can be reduced by storing each part as an equation using an interpolation function.

(12) In the object model means of the time-series image analysis device described in any one of the above items (1) to (11), the periodic component of the motion of the part of the object in the time-series image. And a time series image analysis device which stores only one cycle of parameters.

Therefore, when the motion has periodicity, the amount of model information can be reduced by extracting the cyclic component from the motion and storing only the parameters for one period.

The embodiments relating to the present invention are as follows:
The four embodiments correspond.

(13) In the object model means of the time-series image analysis device described in the above item (1), the possible area calculating means for calculating the possible area of each part regarding the above parameters is added. Characteristic time-series image analysis device.

The embodiment relating to the present invention corresponds to the first embodiment.

Therefore, the parameter of each part of the time series image can be calculated at high speed by calculating the possible area by using the parameter of each part of the target motion. (14) In the object model means of the time-series image analysis device according to (13), the shape of the possible area calculated by the possible area calculating means is held as a shape obtained by combining simple geometrical shapes. A time-series image analysis device characterized by:

The embodiment relating to the present invention corresponds to the first embodiment.

Therefore, by expressing the calculated possible existence area as a combination of parameters of the geometrical shape, it is possible to reduce the information amount of the model.

(15) In the parameter calculating means of the time-series image analysis device described in the item (13), the object in the time-series image is based on the possible area output by the possible-area calculating means. And a candidate area calculating means for calculating the existence candidate area of each part of the time series image analysis apparatus.

The embodiments relating to the present invention correspond to the second and eighth embodiments.

Therefore, by performing the AND processing of the possible area calculated above and the silhouette image of the object in the time series image, and collectively calculating the candidate areas of each part, the processing becomes faster, The post-processing load can be reduced.

(16) In the parameter calculation means of the time-series image analysis device described in the above item (15), the parameter group of each part of the object is calculated based on the possible area output by the candidate area calculation means. A time-series image analysis device for performing.

The embodiment relating to the present invention corresponds to the second embodiment.

Therefore, the load of post-processing can be reduced by determining the connection relation of the candidate regions of the respective parts obtained above and calculating the parameters.

(17) In the parameter calculation means of the time-series image analysis device described in (16) above, when the parameter group is calculated, the candidate area calculated by the candidate area calculation means and the object model are calculated. A time-series image analysis device, wherein an allowable range of each parameter is calculated from the size of each part based on the means.

The embodiment relating to the present invention corresponds to the second embodiment.

Therefore, the load of post-processing can be reduced by calculating the permissible range of the parameters calculated based on the candidate area obtained above and the size of the model.

(18) Item (1) or (17) above
The time-series image analysis device according to any one of paragraphs, further comprising a parameter correction unit that corrects each parameter by comparing with a model in one image of the time-series image. Analyzer.

The embodiment relating to the present invention corresponds to the first embodiment.

Therefore, the difference between the parameters of the model and the time-series image is calculated. If the difference is larger than the threshold value, the parameter is replaced and the matching error is calculated. To calculate
The optimization process becomes faster.

(19) The parameter calculation means of the time-series image analysis device described in any one of the above items (15) to (17) indicates the moving direction of the object extracted by the shape generation means. Moving direction calculating means for calculating, moving direction judging means for judging whether or not the moving direction of the object is the same as the moving direction of the knowledge stored in the object model means, the object and the above Model correction means for correcting the parameters relating to the movement stored in the object model means when the movement directions of the knowledge about the movement stored in the object model means are not the same. Time series image analysis device.

The embodiment relating to the present invention corresponds to the second embodiment.

Therefore, it is possible to analyze an object that moves in both directions with one device.

(20) The matching means of the time-series image analysis device described in the above item (1) extracts all the corresponding parameters calculated from the parameter means, all the parameters of the object model means, and the shape generation means. A time-series image analysis device characterized by collating the silhouette images of the target object thus obtained and minimizing an error.

The embodiment relating to the present invention corresponds to the second embodiment.

(21) In the time series image analysis device described in the item (1), the matching means is further configured as an image representing the object from the parameters of the object calculated by the parameter calculating means. A time-series image analysis device, which corrects the parameters of the object so that the image and the object extracted by the shape generation unit match.

The embodiments relating to the present invention are as follows:
The seven embodiments correspond.

Therefore, by matching the parameters of each part of the object in the time-series image with the parameters of the model and the object in the time-series image, it is possible to perform a more accurate analysis.

(22) The matching means of the time series image analysis device according to any one of the above items (1) and (21) has a synchronizing means for synchronizing the times, and A means is a motion detecting means for detecting a time change of a predetermined motion of each part of the object stored as a parameter in the object model means, and the object at the time detected by the motion detecting means. Time-series image analysis, characterized in that it comprises an object stored in the model means and a time calculation means for calculating the time of the same motion state from the parameters calculated by the parameter calculation means. apparatus.

The embodiments relating to the present invention are:
The embodiments of 5, 9, 10, and 11 correspond.

Therefore, an ellipse image is generated using the parameters of the time of predetermined movement of the model, matching is performed with the silhouette image group of the object in the time series images, and the silhouette with the minimum matching error is generated. By calculating the time of the image, comparison with all the images can be omitted, and thus high-speed processing can be performed.

(23) In the matching means of the time series image analysis device according to the above (12), the operation cycle of the part of the object in the time series image and the object of the previously set object model means. A time-series image analysis device comprising a period calibration means for calibrating the period of the object model means based on the ratio of the operation cycles of the parts.

The embodiments relating to the present invention are as follows:
The embodiments of 5, 9, 10, and 11 correspond.

Therefore, the operation cycle of each time series image is calculated. By calculating the ratio of the calculated time per cycle to the model time per cycle, and by using this ratio to calibrate the model cycle, you can be flexible even when the operating speed of the object changes. Can respond.

(24) In a moving image analysis method for analyzing a predetermined object in a time-series image, the knowledge of the shape of the object and its shape change due to movement is parameterized for each of the parts constituting the object. By modeling by, by storing the knowledge of the movement as a function of time among the modeled parameters, to extract the object from a plurality of images within a predetermined time range in the time-series image, A parameter corresponding to the modeled parameter is calculated, the modeled parameter and the parameter calculated from the object are synchronized, and the shape by the parameter from the synchronized object and the The above modeled parameters are corrected so that the size of the modeled parameters matches the shape, and the A time-series image analysis method characterized in that the parameter calculated based on the above is matched with the parameter calculated from the above function.

The embodiments relating to the present invention correspond to the first and second embodiments.

Therefore, the parameter of each part constituting the object in the time-series image is calculated, the size of the model is corrected based on the extracted object, and the movement of the object and the model are synchronized. After this is taken, the parameters calculated from the object are corrected so that the magnitudes of the parameters calculated from the object and the parameters of the model satisfy a predetermined relationship.

(25) In a moving image analysis method for analyzing a predetermined object in a time-series image, knowledge of the shape and motion of the object is modeled as a parameter relating to a predetermined part, and the model is mathematically stored and stored. , Calculating a rough parameter corresponding to the model for the object, synchronizing the time between the object and the model, calibrating the size between the object and the model, the object parameter A time-series image analysis method, wherein the whole is matched with all the parameters of the model and a predetermined object in the time-series image.

The embodiments relating to the present invention correspond to the first and second embodiments.

Therefore, the parameters of each part of the object in the time-series image are roughly calculated, the size of the object and the model are calibrated, and the synchronization time is calculated. By defining an evaluation function that matches the parameter group of each part, the model parameter group, and the silhouette image of the object in the time-series image, and minimizing this evaluation function, the behavior of the object Can be analyzed with high accuracy.

(26) The synchronizing means described in the above (22) is the time when the area of the object extracted by the shape generating means and the area of the object stored in the object model means becomes minimum or maximum. A time-series image analysis device for calculating.

The embodiment relating to the present invention corresponds to the first embodiment.

(27) The matching means of the time-series image analysis device described in the above item (1) is added with a neural network which outputs parameters of each part of the object when the silhouette image of the shape generation means is input. A time-series image analysis device.

The embodiment relating to the present invention corresponds to the fourth embodiment.

(28) The object model means of the time-series image analysis device according to any one of the above items (1) and (9) to (12) further includes the parameters calculated by the matching means. And a parameter updating means for updating the parameter stored in the object model means based on the correction parameter calculated from the difference between the parameter of the object model means and the parameter of the object model means. .

The embodiment relating to the present invention corresponds to the fifth embodiment.

Therefore, it is possible to improve the accuracy and speed of the processing by adaptively learning the parameters related to the movement of the model as the processing progresses.

(29) The object model means of the time series image analysis device according to any one of the above items (1) and (9) to (12) and (28), Time-series image analysis characterized by hierarchically storing models related to movement of each part, with parameters of arbitrary main parts constituting the upper layer and parameters of parts connecting these parts as the lower layer apparatus.

The embodiment relating to the present invention corresponds to the sixth embodiment.

(30) The object model means of the time-series image analysis device according to the above (29) only includes the parameters of the respective portions whose degree of matching exceeds a predetermined threshold when the matching means performs matching. A time-series image analysis device that is newly changed.

The sixth embodiment corresponds to the embodiment of the present invention.

Therefore, it becomes possible to switch the models only for the necessary parts, and it is possible to perform processing that can handle more situations.

(31) In the object model means of the time series image analysis device according to any one of the above items (1) and (9) to (12) and (28), the matching means is The time series characterized in that the parameters extracted from the object model means for use are parameters selected to have a minimum error with respect to a plurality of movement parameters of the object part. Image analysis device.

The embodiment relating to the present invention corresponds to the seventh embodiment.

Therefore, it is possible to automatically select a model even for a motion in which the motion of the object is unknown.

(32) The parameter calculation means of the time-series image analysis device described in any one of the above items (16) and (17) is a possible area of each part output by the candidate area calculation means. 2. In the time series image analysis apparatus, the parameter group is preferentially calculated from a site in which the area of the possible area is equal to or smaller than a predetermined threshold to limit the possible area of the other area. The eighth embodiment corresponds to the embodiment of the present invention.

Therefore, by adding the processing priority to each part, it is possible to improve the accuracy and speed of the processing.

(33) The matching means of the time-series image analysis device described in any one of the above items (1) and (20) predicts the parameter of the next time of each part from the parameter of each past part. A time-series image analysis device further comprising motion vector predicting means for

The ninth embodiment corresponds to the embodiment relating to the present invention.

Therefore, since the parameters are calculated from the past images as compared with the parameters of the model, the temporal continuity is maintained and good parameters can be obtained even in the optimization with one image.

(34) The matching means of the time-series image analysis device described in any one of the above items (1), (20), and (33) adjusts the adjustment degree of the parameter of each part in each movement. A time-series image analysis device comprising: an adjustment degree calculating means for calculating; and an optimization processing means for updating a minute change given to each part based on the adjustment degree calculated by the adjustment degree calculating means.

The tenth embodiment corresponds to the embodiment of the present invention.

(35) The adjustment degree calculating means of the time series image analysis device according to the above (34) is characterized by calculating the difference between the parameter stored in the object model means and the parameter of the time series image. And time series image analysis device.

Therefore, it is possible to realize high-accuracy and high-speed processing by changing the method of giving a minute change for each part.

The eleventh embodiment corresponds to the embodiment of the present invention.

(36) The matching means of the time-series image analysis device described in any one of the above items (1), (20) and (33) uses the correction information made by the parameter correction means. A time-series image analysis device further comprising certainty factor calculation means for calculating a region and a space in which the parameter of each part in the matching means is slightly changed based on the certainty factor calculation means.

The eleventh embodiment corresponds to the embodiment of the present invention.

Therefore, it is possible to realize high-accuracy and high-speed processing by limiting the range in which a minute change is given to each part.

[0324]

As described above in detail, according to the present invention, by utilizing the knowledge about the shape and the movement of the object, the movement of the object with respect to the part which cannot be analyzed due to the influence of the shielding can be performed. It is possible to provide a time-series image analysis device and an analysis method thereof that can perform analysis with high accuracy.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a schematic overall configuration example of a time-series image analysis device according to a first embodiment of the present invention.

FIG. 2 is a diagram for explaining a model shape stored in an object model unit shown in FIG.

FIG. 3 is a diagram for explaining a possible area in which a model shape stored in an object model unit exists.

FIG. 4 is a diagram showing a specific configuration example of a matching unit shown in FIG.

5 is a diagram showing a concept for explaining a process of a synchronization unit shown in FIG.

FIG. 6 is a diagram showing a concept for explaining a process of a period calibration unit shown in FIG.

7 is a diagram showing a concept for explaining a process of a parameter correction unit shown in FIG.

8 is a diagram showing a concept for explaining a process of an optimization processing unit shown in FIG.

FIG. 9 is a diagram showing a relationship between a processed image and an analysis result in the first embodiment.

FIG. 10 is a diagram showing a specific configuration example of a parameter calculation unit in the second embodiment.

FIG. 11 is a diagram showing a concept in calculating existence candidate regions of respective parts of a target object in a time-series image.

FIG. 12 is a diagram showing a concept in calculating a position, a tilt angle, and an allowable range of a candidate area group having a connection relationship.

FIG. 13 is a conceptual diagram of a marker attachment and a numerical data calculation method.

FIG. 14 is a diagram showing an example of table storage of numerical data.

FIG. 15 is a diagram showing a concept of numerically storing numerical data stored in an object model part for each part by formulating and storing the numerical data.

FIG. 16 is a diagram showing a concept for explaining a parameter calculation of a possible area.

FIG. 17 is a flowchart for explaining synchronization processing.

FIG. 18 is a diagram showing the concept of a time calculation method when the model is in the close state.

FIG. 19 is a flowchart for explaining a period calibration unit.

FIG. 20 is a diagram showing a concept for explaining a process of a period calibration unit.

FIG. 21 is a flowchart for explaining correction of parameters.

FIG. 22 is a flowchart illustrating an optimization process.

FIG. 23 is a diagram showing a concept related to prevention of separation of each part.

FIG. 24 is a flowchart for explaining calculation of parameters.

FIG. 25 is a diagram showing a schematic overall configuration example of a time-series image analysis device as a third embodiment.

FIG. 26 is a diagram showing a configuration of a matching unit in the third embodiment.

FIG. 27 is a diagram showing a configuration example of a matching unit in the time-series image analysis device as the fourth embodiment.

FIG. 28 is a diagram showing a schematic configuration of an optimization processing unit using a hierarchical neural network.

FIG. 29 is a diagram showing a schematic overall configuration example of a time-series image analysis device as a fifth embodiment.

FIG. 30 is a diagram for explaining the concept of updating model parameters.

FIG. 31 is a diagram showing a configuration example of an object model unit.

FIG. 32 is a diagram showing a configuration of a matching unit in the fifth embodiment.

FIG. 33 is a diagram showing a configuration of an object model unit in the time-series image analysis device as the sixth embodiment.

[Fig. 34] Fig. 34 is a diagram illustrating an example of a tree structure of each part with a human being as an example.

FIG. 35 is a diagram showing a schematic overall configuration example of a time-series image analysis device as a seventh embodiment.

FIG. 36 is a diagram showing a configuration of an analysis unit in the seventh embodiment.

[Fig. 37] Fig. 37 is a diagram illustrating a concept regarding a method of selecting an optimal model from a plurality of models for an object in a time-series image.

FIG. 38 is a diagram illustrating a configuration example of a parameter calculation unit of the time-series image analysis device according to the eighth embodiment.

[Fig. 39] Fig. 39 is a diagram illustrating the concept of calculating the existence candidate region of each part of the target object in the time-series image in which the processing priority order is introduced.

FIG. 40 is a diagram showing a concept related to calculation of a limited candidate area 2.

FIG. 41 is a diagram showing a configuration of a matching unit in the time-series image analysis device as the ninth embodiment.

FIG. 42 is a diagram showing the concept of motion vector calculation.

FIG. 43 is a diagram showing the concept of optimization processing using prediction.

FIG. 44 is a diagram showing a configuration of a matching unit in the time-series image analysis device as the tenth embodiment.

FIG. 45 is a diagram showing the concept of adjustment degree calculation.

FIG. 46 is a diagram showing a configuration of a matching unit in the time-series image analysis device as the eleventh embodiment.

FIG. 47 is a diagram showing a concept regarding information obtained by a parameter correction unit.

[Fig. 48] Fig. 48 is a diagram illustrating the concept of how to make a minute change using confidence.

FIG. 49 is a diagram showing a concept related to model inversion.

[Explanation of symbols]

1 ... Input unit, 2 ... A / D unit, 3 ... Buffer memory unit, 4
... auxiliary input / output unit, 5 ... shape generation unit, 6 ... object model unit, 7 ... parameter calculation unit, 8 ... matching unit, 9 ... D / A
Section, 10 ... output section, 11 ... control section.

Claims (3)

[Claims] 1. A time-series image analysis device for analyzing a predetermined object in a time-series image, and shape generation means for extracting the object from a plurality of images within a predetermined time range in the time-series image, Object model means for storing knowledge about the shape and shape change due to movement of the object as a parameter for each part constituting the object, and the object model from the object extracted by the shape generating means. Parameter calculation means for calculating parameters of position and inclination angle type and time corresponding to parameters stored in the means, and the object calculated by the parameter calculation means corresponding to each of the plurality of images A set of parameters whose constituent elements are stored in the model means corresponding to the parameters of the object. Time-series image analyzer, characterized in that it comprises a matching means for matching a set of parameters that the components respectively, a parameter. 2. The matching means further matches an image configured as an image representing the object from the parameters of the object calculated by the parameter calculating means with the object extracted by the shape generating means. The time-series image analysis device according to claim 1, wherein the parameters of the object are corrected so as to do so. 3. A moving image analysis method for analyzing a predetermined object in a time-series image, wherein the knowledge of the shape and shape change of the object is parameterized for each part constituting the object. Of the modeled parameters is stored as a function of time, and the object is extracted from a plurality of images within a predetermined time range in the time-series image to obtain the model. The parameter corresponding to the parameterized parameter is calculated, and the modeled parameter and the parameter calculated from the object are synchronized, and the shape by the parameter from the synchronized object and the modeling Correct the above modeled parameters so that the size of the shape is matched with the shape of the calculated parameters, and calculate based on the object. A time-series image analysis method, characterized in that the obtained parameters are matched with the parameters obtained from the above function.