JP5289290B2 - Posture estimation device - Google Patents

Description

  The present invention relates to a posture estimation apparatus that estimates the posture of a person shown in an image.

  Conventionally, techniques for estimating the posture of a person in an image taken by a camera have been studied. In particular, a technique called shape context matching has been proposed as a technique with excellent robustness for recognizing the shape of an object. And the technique which applied shape context matching to a person's attitude | position estimation is disclosed (for example, refer nonpatent literature 1 and 2).

  A shape context, which is a feature amount used in shape context matching, is calculated as follows. First, a plurality of feature points are set on the contour of the model to be recognized. For the target feature point of the plurality of feature points, a vector from the target feature point to another feature point is obtained, and the angle formed by the reference line such as the X axis and Y axis set for the image and the vector The length of the vector, that is, the distance from the feature point of interest to another feature point is obtained. Then, the circular region centered on the feature point of interest is divided into a plurality of sections in a predetermined angle unit and a predetermined length unit, and the length of the vector and the frequency of the angle to the other corresponding feature point are determined for each section. By calculating, a shape context is obtained. That is, the shape context represents a two-dimensional distribution of the angle and length of a vector connecting the feature point of interest to another feature point. This shape context is obtained for each feature point. Therefore, as many shape contexts as the number of feature points are obtained for one model.

  Shape context matching detects the best match with the set of shape contexts obtained for an unknown model from the set of shape contexts obtained in advance for a plurality of known shape models, and the unknown model is detected. The shape model corresponding to the set.

S.Belongie, J.Malik, and J.Puzicha, "Shape Matching and Object Recognition Using Shape Contexts", IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, 2002, Volume 24, Issue 4, p.509- 522 G.Mori, and J.Malik, "Recovering 3D Human Body Configurations Using Shape Contexts", IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, 2006, Volume 28, Issue 7, p.1052-1062

Even if the joint angle of a part of the human body such as a limb changes, the value of the shape context for the feature point set away from the part does not change greatly. On the other hand, the difference between the two postures, for example, the posture of simply standing and the posture of talking using a mobile phone while standing, may be that the angle of one arm is different.
In this way, if you want to accurately identify multiple postures that differ only in the shape of a part of the human body part, the shape context set obtained for the image of the person to be recognized corresponds to each of these multiple postures. It is necessary to perform matching processing with a set of shape contexts obtained from a plurality of models. Therefore, in the posture estimation method according to the prior art, in order to accurately identify the posture of the person, a great amount of calculation processing is required, and it takes a long time to obtain the posture estimation result. In order to achieve a realistic calculation load, the number of models must be reduced, and as a result, accurate estimation may not be possible. Therefore, there is a possibility that the estimation result of the person's posture cannot be used effectively.

  Therefore, an object of the present invention is to provide a posture estimation device that can accurately estimate the posture of a person shown in an image.

  According to one embodiment of the present invention, a posture estimation device that estimates the posture of a person in an image is provided. Such a posture estimation device detects an edge in an image, sets a plurality of feature points on the edge, and a distribution of positional relationships between each of the plurality of feature points and other feature points A feature amount calculation unit that calculates a shape feature amount indicating a reference feature feature amount that is a shape feature amount obtained in advance for a plurality of reference feature points set in each of a plurality of posture models representing a posture of a person; A storage unit that stores a part parameter value representing a joint angle for each specific part of the posture model corresponding to the reference shape feature quantity, and a reference shape feature quantity similar to each of the shape feature quantities is selected for each feature point A feature parameter selection unit and a part parameter value corresponding to the selected reference shape feature value to obtain a distribution for each specific part and determine whether the part parameter value distribution is concentrated. It has a region shape estimation unit that estimates a shape of the specific part of the person in the image, by using the estimated site shape and a posture estimation unit configured to estimate the posture of the person in the image.

  In such a posture estimation apparatus, the storage unit preferably stores an index representing the direction and distance from the reference feature point to the reference standard point for representing the position of the whole body of the posture model in association with the reference shape feature amount. In this case, the posture estimation apparatus corresponds to an index associated with the reference shape feature value and a shape feature value similar to the reference shape feature value for each of the reference shape feature values selected by the feature value selection unit. A whole body criterion for obtaining a voting position for a reference point representing the position of a person on the image from the position of the feature point, and selecting a reference shape feature quantity having a voting position within a predetermined range from the most frequent position where the voting position is most concentrated It is preferable that the image processing apparatus further includes a point specifying unit, and the partial shape estimation unit uses only the reference shape feature quantity selected by the whole body reference point specification unit among the reference shape feature quantities for estimation of the person's part shape.

  In such a posture estimation apparatus, the storage unit further stores the degree of weight of the posture model in association with the reference shape feature amount, and the whole body reference point specifying unit uses the reference shape feature amount selected by the feature amount selection unit. It is preferable to determine the voting position as a multidimensional value including the position of the reference point on the obtained image and the degree of weight associated with the reference shape feature amount.

  The posture estimation apparatus according to the present invention can accurately estimate the posture of a person shown in an image.

It is a schematic block diagram of the image monitoring apparatus which concerns on one embodiment. (A) is a figure which shows an example of the monitoring image in which the person is reflected, (b) is a figure which shows an example which divided | segmented the monitoring image shown in (a), (c), It is a figure which shows an example of the feature point set on the boundary of the area | region shown by (b). (A) is a figure showing the positional relationship between each section about one feature point set on the outline of a person and other feature points, and (b) is a figure showing an example of a partial shape feature amount It is. It is an example of a search list. It is a figure which shows an example of a reference shape feature-value list | wrist. It is a figure which shows an example of the reference | standard direction vector corresponding to a reference shape feature-value on the basis of each feature point shown by FIG.2 (c). (A) is a figure which shows an example of each parameter contained in a site | part parameter, (b) is a figure which shows the correspondence of a posture model and each parameter. It is a figure which shows an example of an attitude | position determination table. (A)-(c) is a figure which shows an example of the outline of a person's attitude | position, respectively. It is an operation | movement flowchart of an abnormality detection process. It is an operation | movement flowchart of an abnormality detection process.

  Hereinafter, an image monitoring apparatus using an attitude estimation apparatus according to an embodiment will be described with reference to the drawings. This image monitoring apparatus obtains a shape context as a partial shape feature amount representing a distribution of positional relationship with other feature points for a plurality of feature points set on an edge in an image. The image monitoring apparatus compares the partial shape feature amount of each feature point with a partial shape feature amount obtained for a reference feature point set in a specific part such as a limb of a person's posture model. By using only, the shape of the specific part is estimated. In the present embodiment, the shape of the part is represented by the angle of the joint of the limb. For example, the arm has joints at the shoulder and elbow, the direction of the upper arm with respect to the shoulder joint is represented by three angles, and the direction of the forearm with respect to the upper arm is represented by one angle corresponding to the angle of the elbow joint. The Therefore, the shape of the arm is specified with a total of four angles as parameters. Similarly, in the leg, the direction of the thigh with respect to the hip joint is represented by three angles, and the direction of the lower leg with respect to the thigh is represented by one angle corresponding to the angle of the knee joint. Therefore, the shape of the leg is also specified with a total of four angles as parameters. The image monitoring apparatus estimates the posture of the whole person by combining the shapes of the parts.

  FIG. 1 is a schematic configuration diagram of an image monitoring apparatus 1 according to one embodiment. The image monitoring device 1 includes an imaging unit 2 and a posture estimation device 3. The posture estimation device 3 includes an interface unit 4, a storage unit 5, an output unit 6, and a control unit 7.

The imaging unit 2 forms a two-dimensional detector composed of a photoelectric converter having sensitivity to visible light or near-infrared light, such as CCD or C-MOS, and an image of a monitoring region on the two-dimensional detector. And an imaging optical system. The imaging unit 2 can be, for example, a camera that continuously shoots according to the NTSC standard. Alternatively, the imaging unit 2 may generate a higher resolution image such as so-called high vision. Then, the imaging unit 2 generates a monitoring image obtained by capturing the monitoring area, for example, as a grayscale image or a color image in which the luminance of each pixel is represented by 256 gradations.
The image output of the imaging unit 2 is connected to the interface unit 4 of the posture detection device 3, and the imaging unit 2 outputs the generated monitoring image to the posture estimation device 3 every time a monitoring image is generated.

The interface unit 4 of the posture estimation device 3 includes an interface that connects the posture estimation device 3 and the imaging unit 2 and a control circuit thereof. The interface unit 4 has a configuration applied to an image communication standard that the imaging unit 2 complies with. The interface unit 4 may include an analog-digital conversion circuit that converts the monitoring image into a digital image when the monitoring image is generated as an analog image.
The interface unit 4 is connected to the control unit 7 and passes the monitoring image received from the imaging unit 2 to the control unit 7.

The storage unit 5 of the posture estimation device 3 includes a storage device such as a nonvolatile semiconductor memory such as a flash memory, a volatile semiconductor memory, or a magnetic disk (HDD).
The storage unit 5 stores various programs and data used in the image monitoring apparatus 1. In addition, the storage unit 5 may store, as a background image, for example, a monitoring image in which an intruder is not captured, which is acquired from the imaging unit 2 when the image monitoring device 1 is activated or periodically.
In addition, the storage unit 5 represents a distribution of positional relationships with other reference feature points obtained for reference feature points set on a contour of the posture model, more preferably on a part such as a limb of the person's posture model. The reference shape feature value is stored. Further, the storage unit 5 stores a reference shape feature quantity list indicating the reference shape feature quantity and the position of a reference standard point representing the position of the whole body of the posture model viewed from the reference feature point corresponding to the reference shape feature quantity. Furthermore, the storage unit 5 stores a posture determination table that represents the relationship between the shape of each part of the posture model and the posture of the person, and the determination flag that indicates whether the posture of the person is normal or abnormal. Details of these data stored in the storage unit 5 will be described later.

  The output unit 6 of the posture estimation apparatus 3 includes a communication interface connected to a communication network such as a local area LAN or a public line network and a control circuit thereof. The output unit 6 receives an abnormality detection signal indicating that an abnormality has been detected by the control unit 7 from the control unit 7, and the abnormality detection signal is connected to the image monitoring apparatus 1 via a communication network. You may output to an apparatus or a monitoring center apparatus. The output unit 6 also receives from the control unit 7 the monitoring image acquired at and after the abnormality detection, the estimated name of the posture of the person, etc. from the control unit 7 together with the abnormality detection signal. Or the like may be output to the security device or the monitoring center device.

The control unit 7 of the posture estimation device 3 includes, for example, one or a plurality of microprocessor units and their peripheral circuits. The control unit 7 controls the entire image monitoring apparatus 1. Further, the control unit 7 estimates the posture of the person in the monitoring area based on the monitoring image received from the imaging unit 2 via the interface unit 4 and the data stored in the storage unit 5. And the control part 7 determines whether abnormality generate | occur | produced based on the estimated attitude | position.
Therefore, the control unit 7 includes a feature point setting unit 71, a partial shape feature amount calculation unit 72, a similarity calculation unit 73, a feature amount selection unit 74, a whole body reference point specifying unit 75, and a part shape estimation unit. 76, a posture estimation unit 77, and a determination unit 78. Each of these units included in the control unit 7 is mounted as a functional module of a program that operates on the microprocessor unit, for example.

The feature point setting unit 71 sets a plurality of feature points that serve as reference points for calculating a partial shape feature amount, which will be described later, on the edge including the outline of the person shown in the monitoring image.
For this purpose, for example, the feature point setting unit 71 performs segmentation processing on the monitoring image and divides the monitoring image into a plurality of regions having similar features. The feature point setting unit 71 can use, for example, a mean shift segmentation method as the segmentation process. This average shift division method is described in, for example, D. Comaniciu, P. Meer, "Mean Shift: A Robust Approach. Toward Feature Space Analysis", IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, 2002, Volume 24, Issue 5, p. 603-619.

  The feature point setting unit 71 sets the boundary of the region as an edge, and sets a predetermined number of feature points on the boundary. Note that the number of feature points to be set is determined according to the shooting conditions or the processing capability of the control unit 7. For example, the number of feature points can be 25, 50, or 100.

The feature point setting unit 71 sequentially assigns numbers to the pixels representing the boundary of the region, generates a random number, and sets the pixels assigned the number corresponding to the random number as the feature points.
Alternatively, the feature point setting unit 71 sets, for example, a pixel on the boundary closest to a predetermined reference position (for example, the upper left or the center of the monitoring image) as a feature point, and is separated from the feature point by a randomly determined distance. The pixel on the boundary at the specified position is set as the next feature point. Thereafter, the feature point setting unit 71 repeatedly executes the same process until the number of set feature points reaches a predetermined number. At that time, the feature point setting unit 71 may set the feature point so that the distance between the feature point to be newly set and the already set feature point is more than a predetermined distance (for example, 5 pixels or more). Good.

  The feature point setting unit 71 may set a larger number of feature points as the density of the boundary between regions is higher. In this case, for example, the feature point setting unit 71 divides the monitoring image into 4 horizontal sections × 4 vertical sections, and counts the number of pixels located on the boundary of the area included in each section. Then, for each section, the feature point setting unit 71 multiplies the value obtained by dividing the number of pixels on the boundary included in the section by the total number of pixels on the boundary of the entire monitoring image by the total number of feature points to be set. Determine the number of feature points for each section. And the feature point setting part 71 sets a feature point similarly to the above about each division.

FIG. 2A is a diagram illustrating an example of a monitoring image in which a person is shown, and FIG. 2B is a diagram illustrating an example in which the monitoring image illustrated in FIG. 2A is divided into regions. FIG. 2C is a diagram illustrating an example of feature points set on the boundary of the region illustrated in FIG.
In the monitoring image 200 shown in FIG. 2A, a single person 201 standing upright is shown indoors. In FIG. 2 (b), the boundary of each region is represented by a solid line. As shown in FIG. 2B, the monitoring image 200 is divided into a plurality of areas, so that the area including the person 201 is distinguished from other areas. For example, in the area 210, the right hand part of the person is shown, and is distinguished as an area different from other parts. In FIG. 2C, a plurality of points 211 represent feature points set on the boundary of each region. In FIG.2 (c), the feature point is set equally on the boundary of each area | region. In the present embodiment, only information relating to feature points set on the boundary corresponding to the outline of the person is selected by processing described later. Therefore, a feature point may be set at a place other than the outline of the person like a point 211a shown in FIG.

  Alternatively, the feature point setting unit 71 performs an edge detection process on the monitoring image by using an edge detection filter such as a sobel filter or a Laplacian filter instead of dividing the monitoring image into regions. Edges may be extracted. Also in this case, the feature point setting unit 71 performs the same processing as that for setting the feature points on the region boundary obtained by dividing the region, so that the edge having an edge strength stronger than the predetermined edge strength is obtained. A predetermined number of feature points are set on the top.

Furthermore, the feature point setting unit 71 performs background difference processing between the monitoring image and a background image that is an image of the monitoring area when the person to be detected is not captured in the monitoring area, so that the person is captured. A person area that is a possible area may be extracted. The feature point setting unit 71 sets a predetermined number of feature points on the contour of the person region as an edge. In this case, the feature point setting unit 71 obtains the contour length of the person area by, for example, counting the number of contour pixels of the person area. And the feature point setting part 71 determines the distance between the feature points showing the space | interval of adjacent feature points by dividing the outline length by the predetermined number of feature points.
The feature point setting unit 71 sets an arbitrary contour pixel on the contour of the person region as the first feature point. The feature point setting unit 71 responds in order from the first feature point in the clockwise or counterclockwise direction along the outline of the person area, each time the feature point is separated from the feature point set immediately before by the distance between the feature points. The contour pixel is set as the next feature point.
The feature point setting unit 71 passes the position coordinates of the set feature points to the partial shape feature amount calculation unit 72.

  The partial shape feature value calculation unit 72 calculates a partial shape feature value representing the feature of the contour shape of a specific part of the human body for each feature point. In the present embodiment, the partial shape feature quantity calculation unit 72 calculates a shape context as the partial shape feature quantity. That is, the partial shape feature amount calculation unit 72 obtains a vector from the feature point of interest to another feature point, an angle θ that the vector forms with a reference line such as an X axis or a Y axis defined for the image, and the angle θ The length of the vector, that is, the distance r from the feature point of interest to another feature point is obtained. Then, the partial shape feature quantity calculation unit 72 divides the circular region centered on the feature point of interest into a plurality of sections in a predetermined angle unit and a predetermined length unit, and for each section up to the corresponding other feature point. The frequency of the vector length r and the angle θ, that is, the number of other feature points existing in the section is calculated. Then, the partial shape feature amount calculation unit 72 obtains the shape context by normalizing the frequency of each section by dividing it by the area of the section. In the present embodiment, the reference line is a horizontal line that goes to the right from the feature point of interest. The predetermined angle unit was 30 °, and the predetermined length was 10 pixels. Note that the predetermined length unit may be determined so that the logarithm of the distance corresponding to the boundary of the section is equally spaced, for example, so as to increase as the distance from the target feature point increases. Further, the predetermined length unit may be determined so that the areas of the sections are equal.

FIG. 3A is a diagram showing the positional relationship between each section of one feature point 301 set on the person's outline 300 and other feature points, and FIG. 3B is a partial shape. It is a figure which shows an example of the feature-value. In FIG. 3A, each of the concentric and radially segmented partial annular regions centering on the feature point 301 corresponds to one section. For example, since the section 302 includes two feature points 303 and 304, the frequency of the section 302 is 2. On the other hand, since the section 305 includes one feature point 306, the frequency of the section 305 is 1.
As shown in FIG. 3B, the partial shape feature quantity 310 is a normalized two-dimensional histogram of the angle θ and the distance r, the horizontal axis represents the angle θ, and the vertical axis represents the distance r. The height axis represents the normalized frequency. In this example, 12 sections are set for the angle θ and 6 sections for the distance r. The normalized frequency 311 of each section is represented as a solid bar graph.
The partial shape feature amount calculation unit 72 passes the position coordinates of each feature point and the partial shape feature amount obtained for each feature point to the similarity calculation unit 73.

  The similarity calculation unit 73 calculates the similarity between each partial shape feature amount obtained for each feature point and each of the plurality of reference shape feature amounts stored in the storage unit 5. Here, each reference shape feature amount is set on a posture model representing a specific posture of a person, more preferably a plurality of feature points on its contour, and a specific part (for example, arm or leg) of the posture model. ) Is a shape context obtained by a method similar to that of the partial shape feature amount calculation unit 72. Therefore, similarly to the partial shape feature quantity, the reference shape feature quantity also includes the distance r from the target reference feature point to another reference feature point, and the angle θ formed by the vector connecting the two reference feature points and the reference line. It is a two-dimensional histogram showing the normalized frequency for every section. Further, a plurality of reference shape feature amounts are obtained for each of a plurality of posture models representing different postures. The reference shape feature quantity is stored in advance in the storage unit 5 together with the identification number. The reference shape feature value is calculated separately from the model synthesized on the image, and which part of the model the reference shape feature value corresponds to which reference feature point by the simulation etc. May be determined.

The similarity calculation unit 73 determines the similarity based on the Manhattan distance calculated by regarding each normalized frequency of the two-dimensional histogram included in the reference shape feature quantity and the partial shape feature quantity as one element, for example. . That is, the similarity calculation unit 73 calculates the similarity s _ij of the partial shape feature quantity p _i for the i-th feature point and the j-th reference shape feature quantity r _j stored in the storage unit 5 according to the following equation. calculate.
N is the total number of sections included in the partial shape feature amount. P _i = (p _i1 , p _i2 , ..., p _iN ) represents the partial shape feature quantity corresponding to the i-th feature point, and its element p _ik (k = 1,2, ..., N) represents the normalized frequency of the kth section. The section order may be determined arbitrarily. R _j = (r _j1 , r _j2 ,..., R _jN ) represents the j-th reference shape feature amount, and its element r _jk represents the normalized frequency of the k-th section. The obtained similarity is a value included in the range of 0 to 1, and becomes 1 when the partial shape feature quantity p _i and the reference shape feature quantity r _j completely match.
The similarity calculation unit 73 sets the distance between the partial shape feature value and the reference shape feature value to another distance such as an intersection (the sum of the minimum values of the two frequencies in the corresponding section) or Earth Mover's Distance. You may calculate by a scale.
The similarity calculation unit 73 passes the position coordinates of each feature point, the similarity obtained for each feature point, and the identification number of the reference shape feature amount corresponding to the similarity to the feature amount selection unit 74.

The feature amount selection unit 74 selects a predetermined number of reference shape feature amounts in order from the highest similarity for each feature point. The predetermined number can be set to 3, 5, or 10, for example.
Alternatively, the feature amount selection unit 74 may refer to the storage unit 5 and select a reference shape feature amount having a similarity equal to or higher than a predetermined threshold for each feature point. The predetermined threshold is, for example, a distribution of similarity calculated for a plurality of feature points known to exist on the outline of a person, and a similarity calculated for a plurality of feature points set other than the outline of the person. By performing discriminant analysis on the distribution, the two similarity distributions are determined in advance as threshold values that can be most separated. For example, the predetermined threshold is set to a value of about 0.5 to 0.7.
For each feature point, the feature quantity selection unit 74 creates a search list including the position coordinates, the similarity of the selected reference shape feature quantity, and the identification number, and stores the search list in the storage unit 5.

  FIG. 4 shows an example of the search list. In the search list 400, in order from the left column, each column has a feature point identification number 401 set on the monitoring image, a position coordinate 402 of the feature point on the monitoring image, and the feature point. The identification number 403 of the selected reference shape feature amount and the similarity 404 with respect to the selected reference shape feature amount are displayed. For example, referring to the search list 400, for the feature point with the identification number “01”, the position coordinates of the feature point are (100, 150), and there are three identification numbers “123”, “456”, and “789”. The reference shape feature quantities are selected, and the similarities to the selected reference shape feature quantities are 0.8, 0.7, and 0.85, respectively.

  The whole body reference point specifying unit 75 specifies a reference point representing the position of the whole body of the person on the monitoring image from the reference shape feature amount registered in the search list, and a feature point corresponding to the reference shape feature amount A reference shape feature quantity is selected whose positional relationship with the reference point is substantially the same as the positional relation between the reference feature point corresponding to the reference shape feature quantity and the reference reference point. Therefore, the whole body reference point specifying unit 75 uses a vector from the reference feature point corresponding to the reference shape feature amount selected for each feature point to the reference reference point representing the whole body position of the posture model. The reference reference point is set, for example, at the barycentric position of the posture model in which the reference feature point is set. Alternatively, the reference reference point is a point that can represent the position of the whole body of the posture model, such as a point located in the body of the posture model, such as a point corresponding to the navel of the posture model, other than the center of gravity position of the posture model. I just need it.

  The whole body reference point specifying unit 75 refers to the search list and specifies the identification number of the reference shape feature value selected for each feature point. Then, the whole body reference point specifying unit 75 refers to the reference shape feature value list stored in the storage unit 5 to obtain a vector from the reference feature point corresponding to the reference shape feature value to the reference reference point.

  FIG. 5 is a diagram illustrating an example of the reference shape feature amount list. In the reference shape feature quantity list 500, in order from the left column, each column corresponds to a reference shape feature quantity identification number 501, a whole body parameter 502 corresponding to the reference shape feature quantity, and the reference shape feature quantity, respectively. A part parameter 503 representing the angle of the joint included in the limb is represented as the shape of the limb to be performed.

  The whole body parameter 502 includes a weight 511 that represents the degree of weight of the posture model in which a reference feature point corresponding to the reference shape feature value is set, and a standard that is an index that represents the direction and distance of the reference reference point from the reference feature point. A direction vector 512 is included. The degree of fatness 511 is, for example, 1.0 for a posture model corresponding to a standard body shape, and is expressed as a ratio of the diameter of the body portion of the posture model of interest to the diameter of the body portion of the posture model of the standard body shape. The The diameters of the arms and legs are changed according to the ratio of the trunk diameters. Each combination may be considered by making the ratio of the diameter of the trunk, the ratio of the diameter of the arm, and the ratio of the diameter of the legs independently variable. Accordingly, the fatness degree 511 has a smaller value for a posture model that is more lean than the standard body shape, and conversely has a larger value for a posture model that is more fat than the standard body shape. For example, the posture model is set so that the degree of fatness 511 is included in the range of 0.8 to 1.2 and is increased or decreased by 0.01 units.

  The site parameter 503 includes a right arm parameter 521 representing the angle of the right shoulder joint and the angle of the right elbow, a left arm parameter 522 representing the angle of the left shoulder joint and the left elbow joint, and the angle of the right hip joint. A right leg parameter 523 representing the angle of the right knee joint and a left leg parameter 524 representing the angle of the left hip joint and the angle of the left knee joint are included. Details of the site parameter will be described later.

  The whole body reference point specifying unit 75 is selected for each reference shape feature amount registered in the search list as having a similar reference shape feature amount to the corresponding reference direction vector indicated in the reference shape feature amount list. The position of the reference point in the coordinate system of the monitoring image is obtained by adding the position coordinates of the feature points.

  FIG. 6 is a diagram illustrating an example of a reference direction vector corresponding to the reference shape feature quantity selected by the feature quantity selection unit 74 based on each feature point shown in FIG. In FIG. 6, points 601 to 603 each represent a feature point. In addition, arrows 601a to 601e respectively represent a base direction vector corresponding to the reference shape feature amount selected for the feature point 601 with the feature point 601 as the origin. Similarly, arrows 602a to 602e and arrows 603a to 603e represent reference direction vectors corresponding to the reference shape feature quantities selected for the feature points 602 and 603, respectively, with the feature points 602 and 603 as the origin. It is.

If the feature point and the reference feature point corresponding to the reference shape feature amount selected for the feature point are feature points set in a similar posture and close to the same part, the feature point The reference point obtained with respect to represents a position substantially the same as the position of the reference reference point with respect to the posture model. For example, in such a case, if the reference reference point is set on the navel of the posture model, the reference point is also located near the navel of the person image on the monitoring image. Therefore, for each feature point, if the reference shape feature quantity of the reference feature point set in the same part on the posture model having the same posture as the posture of the person shown in the monitoring image is selected, it is shown in FIG. As described above, the positions of the reference points calculated for the plurality of feature points (that is, the positions ahead of the reference direction vectors directed from the plurality of feature points) are substantially the same. Conversely, for each feature point, a reference feature point set in a posture model that has a posture different from the posture of the person in the monitoring image or a feature model that is set in a posture model that has the same posture as that person If the reference shape feature amount of the reference feature point set in a part different from the point is selected, the positions of the reference points calculated for each feature point are different from each other.
Therefore, for example, a reference feature point corresponding to a reference shape feature amount where the tip of the reference direction vector is located in an area where the tip of the reference direction vector is most concentrated, such as a region 604 in FIG. There is a high possibility that the feature point is set to the posture model having the same posture as the person in the image and is attached to the same part as the corresponding feature point.

  Therefore, the whole body reference point specifying unit 75 specifies the region where the reference points obtained for each reference shape feature amount are most concentrated as the most frequent region. Then, the whole-body reference point specifying unit 75, among the reference shape feature amounts registered in the search list, is a voting position obtained by adding the position coordinates of the feature point to the corresponding reference direction vector and a position representing the most frequent region. A reference shape feature quantity whose distance from a certain most frequent position is within a predetermined distance is extracted, and other reference shape feature quantities are deleted from the search list.

For example, the whole body reference point specifying unit 75 divides the monitoring image into a plurality of blocks in order to determine the most frequent region. Then, the whole body reference point specifying unit 75 has the highest frequency that includes a voting position obtained by adding the position coordinates of the feature point to a reference direction vector corresponding to the reference shape feature amount registered in the search list among the plurality of blocks. Let the region be the most frequent region. In this case, each block can be, for example, an area having a width of 5 pixels in the horizontal direction and the vertical direction.
Alternatively, the whole body reference point specifying unit 75 may check the frequency at which the voting position is included in the search area while moving the search area having a predetermined shape and size on the monitoring image. In this case, the whole body reference point specifying unit 75 sets the search area at the position where the frequency is the highest as the most frequent area. The search area can be, for example, a circle with a radius of 5 pixels.

  Further, the whole body reference point specifying unit 75 may determine the most frequent region in a three-dimensional space represented by the horizontal and vertical coordinates on the monitoring image and the degree of weight. Accordingly, the whole body reference point specifying unit 75 can also take into account variations in the partial shape feature amount due to a change in the positional relationship between the feature points according to the body shape of the person shown in the monitoring area. Therefore, the whole body reference point specifying unit 75 can more accurately select the reference shape feature amount corresponding to the reference feature point set in the posture model having the same posture as the posture of the person shown in the monitoring image.

In this case, the whole body reference point specifying unit 75 corresponds to the two-dimensional coordinate value obtained by adding the position coordinates of the feature point to the reference direction vector corresponding to the reference shape feature value registered in the search list, and the reference shape feature value. The voting position is represented by three values of the fatness degree. The whole-body reference point specifying unit 75 divides a three-dimensional space having axes that are orthogonal to the horizontal and vertical directions of the monitoring image and the degree of weight, into blocks of a predetermined size, and the frequency of the voting position for each block Ask for. The whole body reference point specifying unit 75 sets the block with the highest vote count as the most frequent region. In this case, for example, each block can be a three-dimensional space having a width of 5 pixels in each of the horizontal direction and the vertical direction and a width of 0.05 with respect to the thickness.
In addition, the whole body reference point specifying unit 75 moves the search area having a predetermined shape and size, and determines the position of the search area where the frequency of the voting position included in the search area is the highest, so that The region may be determined.

  The whole body reference point specifying unit 75 sets the center of the most frequent area or the center of gravity of the voting position included in the most frequent area as the most frequent position. Then, the whole body reference point specifying unit 75 calculates the distance between the vote position and the most frequent position obtained for each of the reference shape feature values registered in the search list. Then, the whole body reference point specifying unit 75 deletes the reference shape feature quantity whose calculated distance is larger than a predetermined threshold from the search list. The threshold value is, for example, the radius of the search region for determining the most frequent region or the same value as the distance from the center of the block to the end point, or the radius of the search region or the distance from the center of the block to the end point, 0.5 It can be a value multiplied by ~ 2.

  The part shape estimation unit 76, based on the part parameter corresponding to the reference shape feature amount selected by the whole body reference point specifying unit 75, the shape of each part such as an arm or a leg, that is, the angle of the joint included in each part Is estimated.

FIG. 7A is a diagram illustrating an example of each parameter included in the part parameter, and FIG. 7B is a diagram illustrating a correspondence relationship between the posture model and each parameter.
Since the degree of freedom of the left and right shoulder joints and the left and right hip joints is 3, the angles of these joints are represented by three parameters. Further, since the degree of freedom of the left and right elbow joints and the left and right knee joints is 1, the angles of those joints are represented by one parameter. Therefore, as shown in FIG. 7A, the right arm parameter 701 has three parameters αr, βr, and γr representing the angle of the right shoulder joint and one parameter φr representing the angle of the right elbow joint. . Similarly, the left arm parameter 702, the right leg parameter 703, and the left leg parameter 704 each have four parameters representing an angle.

In FIG. 7B, a coordinate system 711 indicates a coordinate system corresponding to the parameters of the right shoulder joint. In this embodiment, the angle formed by the upper right arm and the vertical axis of the posture model is αr, the angle formed by the horizontal axis of the posture model is βr, and the posture model is orthogonal to the vertical and horizontal axes of the posture model. Let γr be the angle formed by the positive axis in the direction toward the front.
A coordinate system 712 represents a coordinate system for parameters of the right elbow joint. In the present embodiment, the angle formed by the right forearm and the upper right arm is φr.
A coordinate system 713 indicates a coordinate system corresponding to the right hip joint parameter. In this embodiment, the angle formed by the right thigh with the vertical axis of the posture model is ψr, the angle formed with the horizontal axis of the posture model is ηr, and is orthogonal to the vertical and horizontal axes of the posture model. Let ζr be an angle formed with an axis that is positive in the direction toward the front of.
A coordinate system 714 represents a coordinate system for parameters of the right knee joint. In the present embodiment, an angle formed by the right thigh and the right lower leg is assumed to be ξr.
In addition, the angle of the joint included in the part parameter with respect to the left half is determined based on a coordinate system obtained by inverting the coordinate system for the right half to the left and right.

The part shape estimation unit 76 aggregates part parameters corresponding to each of the reference shape feature values registered in the search list for each predetermined angle range, and four parts for each of the right arm, the left arm, the right leg, and the left leg. A frequency distribution is obtained in a four-dimensional parameter space defined from parameters. Then, the part shape estimation unit 76 determines the representative value of the angle range to which the mode value of the frequency belongs as the posture of the part. Thereby, the part shape estimation part 76 has the joint angle for each part of the person shown in the monitoring image, that is, the part shape of each limb such that the right arm is extended upward or the right leg is bent. Can be estimated. The representative value can be, for example, an average value of the angle parameters of the part parameters included in the angle range to which the mode value belongs, or an angle value representing the center of gravity of the angle range to which the mode value belongs.
In addition, the method used for the frequency calculation can be the same method as the method used by the whole-body reference point specifying unit 75 to determine the most frequent region, for example. The angle unit for each joint angle parameter may be, for example, 5 °, 10 °, or 30 °.
The part shape estimation unit 76 notifies the posture estimation unit 77 of the angle of the joint of each part.

The posture estimation unit 77 estimates the posture of the person shown in the monitoring image based on the joint angle of each part determined by the part shape estimation unit 76.
In the present embodiment, the posture estimation unit 77 refers to a posture determination table that associates the angle ranges of the joints of each part corresponding to each posture of the person.

FIG. 8 is a diagram illustrating an example of the posture determination table. In the posture determination table 800, in order from the left column, each column includes an identification number 801 for uniquely identifying the posture of the entire person, and the joint angle range 802 corresponding to the posture of the entire person. , A posture name 803 of the whole person and a determination flag 804 indicating whether the posture is normal or abnormal are displayed. An angle range 802 of each joint indicates a range of four angle parameters for each limb, that is, a total of 16 angle parameter ranges.
The posture estimation unit 77 determines, for each posture specified by the posture identification number, whether or not the angle parameter of the joint of each part is included in the angle range defined for each joint. Then, when all angle parameters are included in the corresponding angle range determined for each joint, the posture estimation unit 77 represents the posture of the person shown in the monitoring area by an identification number corresponding to the angle range. Estimated to be posture.
The posture estimation unit 77 passes the estimated posture identification number to the determination unit 78.

The determination unit 78 determines whether an abnormality has occurred based on the posture identification number received from the posture estimation unit 77. Specifically, the determination unit 78 refers to the posture determination table, and the determination flag corresponding to the posture identification number received from the posture estimation unit 77 is a value indicating normal or abnormal. Check whether the value indicates that.
If the determination unit 78 indicates that the value of the determination flag is normal, the determination unit 78 determines that there is no abnormality in the person shown in the monitoring image. On the other hand, if the determination unit 78 indicates that the value of the determination flag is abnormal, the determination unit 78 determines that an abnormality has occurred in the person shown in the monitoring image.
Note that the determination unit 78 determines that an abnormality has occurred in the person shown in the monitoring image when the above determination result becomes abnormal for each monitoring image acquired continuously during a certain period of time. May be.

FIGS. 9A to 9C are diagrams each showing an example of a contour of a person's posture.
A posture outline 901 shown in FIG. 9A represents a posture in which a person is walking. A posture outline 902 shown in FIG. 9B represents a posture in which a person stops and makes a call using a mobile phone. A posture outline 903 shown in FIG. 9C represents a posture in which a person raises both hands. This posture corresponds to, for example, a posture that is taken when a person is hit with a weapon by another person. Therefore, when the estimated posture of the person is the posture shown in FIG. 9C, the image monitoring apparatus 1 can determine that some abnormality has occurred in the person shown in the monitoring image. On the other hand, when the estimated posture of the person is the posture shown in FIG. 9A or 9B, the image monitoring apparatus 1 determines that there is no abnormality in the person shown in the monitoring image. it can. Even if the posture of the person shown in the monitoring image is the same, the criterion for determining whether the person is normal or abnormal differs depending on the set monitoring area. Therefore, the value of the determination flag in the posture determination table is preferably set according to the monitoring area.

  When the determination unit 78 determines that an abnormality has occurred in the monitoring area, the control unit 7 transmits an abnormality detection signal to the output unit 6 and causes the output unit 6 to issue an alarm.

The operation of the abnormality detection process including the attitude estimation process will be described with reference to the flowcharts shown in FIGS. This operation is controlled by the control unit 7 of the posture estimation device 3. The following operations are repeated at the monitoring image capturing interval.
When the abnormality detection process is started, the control unit 7 acquires a monitoring image of the monitoring area captured by the imaging unit 2 via the interface unit 4 (step S101). Next, the feature point setting unit 71 of the control unit 7 sets a plurality of feature points on the edge detected from the monitoring image (step S102).

Thereafter, the partial shape feature value calculation unit 72 of the control unit 7 calculates a partial shape feature value for each feature point (step S103).
Next, the control unit 7 sets the i-th feature point as the feature point of interest (step S104). Note that i is set to 1 for the feature point selected first. The control unit 7 reads the reference shape feature amount corresponding to the jth reference feature point from the storage unit 5 (step S105). Note that j is set to 1 with respect to the reference shape feature amount read first. Then, the similarity calculation unit 73 calculates the similarity Sij between the partial shape feature quantity of the i-th feature point and the j-th reference shape feature quantity (step S106).

The feature amount selection unit 74 of the control unit 7 determines whether or not the similarity Sij obtained by the similarity calculation unit 73 is greater than a predetermined threshold L (step S107). When the similarity Sij is larger than the threshold value L, the feature amount selection unit 74 determines that the partial shape feature amount calculated for the i-th feature point is similar to the j-th reference shape feature amount. Then, the feature quantity selection unit 74 registers the identification number of the j-th reference shape feature quantity in the search list in association with the i-th feature point (step S108).
In addition, the whole body reference point specifying unit 75 of the control unit 7 among the whole body parameters corresponding to the jth reference shape feature amount, the partial shape feature amount determined to be similar to the reference direction vector and the jth reference shape feature amount. The voting position is calculated from the position coordinates of the i-th feature point having (step S109).

After step S109 or when the similarity Sij is equal to or smaller than the threshold value L in step S107, the control unit 7 determines whether j is smaller than the total number of reference shape feature amounts stored in the storage unit 5. (Step S110). When j is smaller than the total number of reference shape feature values, the control unit 7 increments j by 1 (step S111). And the control part 7 repeats the process of step S105-S110.
On the other hand, when j is equal to the total number of reference shape feature amounts, the control unit 7 determines whether i is smaller than the total number of feature points set (step S112). When i is smaller than the total number of feature points, the control unit 7 increments i by 1 (step S113). And the control part 7 repeats the process of step S104-S112.

  As shown in FIG. 11, the whole-body reference point specifying unit 75 includes a plurality of blocks obtained by dividing the monitoring image or a block having the highest frequency of voting positions included in a search area set while moving on the monitoring image. Alternatively, the position of the search area is determined as the most frequent area. Then, the whole body reference point specifying unit 75 determines the centroid of the voting position included in the most frequent area or the center of the most frequent area as the most frequent value position (step S114).

When the most frequent value position is determined, the control unit 7 sets the k-th reference shape feature value registered in the search list as the feature value of interest (step S115). Note that k is set to 1 for the reference shape feature amount registered at the beginning of the search list.
Thereafter, the whole body reference point specifying unit 75 calculates the distance d between the voting position corresponding to the feature of interest and the most frequent value position (step S116). Then, the whole body reference point specifying unit 75 determines whether or not the distance d is greater than a predetermined threshold value K (step S117). When the distance d is larger than the threshold value K, the whole body reference point specifying unit 75 deletes the target feature amount from the search list (step S118). On the other hand, when the distance d is equal to or less than the threshold value K, the part shape estimation unit 76 of the control unit 7 uses a parameter representing the angle of the joint included in the part parameter corresponding to the feature amount of interest in the four-dimensional parameter space. The frequency of the corresponding angle range is incremented by 1 (step S119).
After step S118 or S119, the control unit 7 determines whether k is smaller than the total number of reference shape feature amounts registered in the search list (step S120). If k is smaller than the total number of reference shape feature amounts registered in the search list, the control unit 7 increments k by 1 (step S121). And the control part 7 repeats the process of step S116-S120.

  On the other hand, when k is equal to the total number of reference shape feature amounts registered in the search list, the part shape estimation unit 76 estimates the joint angle of the part based on the angular range having the highest frequency for each part ( Step S122). Then, the posture estimation unit 77 of the control unit 7 estimates the posture corresponding to the combination of the joint angles of each part as the posture of the person shown in the monitoring area (step S123).

Next, the determination unit 78 of the control unit 7 refers to the posture determination table stored in the storage unit 5 to determine whether or not the estimated posture is abnormal (step S124). When the estimated posture is normal, the control unit 7 ends the abnormality detection process.
On the other hand, when the estimated posture is abnormal, the determination unit 78 determines that some abnormality has occurred with respect to the person in the person candidate area. And the determination part 78 outputs an abnormality detection signal to the output part 6, and issues a warning (step S125).
After step S125, the control unit 7 ends the abnormality detection process.

As described above, the image monitoring apparatus using the posture estimation device according to the embodiment of the present invention specifies the partial shape feature amount obtained for each feature point from the posture model such as a person's limb. Only the reference partial shape feature amount similar to the partial shape feature amount is selected as compared with the reference partial shape feature amount obtained for the reference feature point set in the part. This reference feature point is preferably set to the contour of the posture model. The image monitoring apparatus estimates the shape of each part represented by the selected reference partial shape feature amount, that is, the shape of each part of the person shown in the monitoring image, based on the joint angle. The image monitoring apparatus estimates the posture of the person's whole body by combining the shapes of the estimated parts. As described above, since this image monitoring apparatus can estimate the shape of each part, it is a case where the posture of the person shown in the monitoring image must be distinguished from the posture in which only part of the part is different in shape. Can also be estimated accurately. Further, this image monitoring apparatus further selects the reference shape feature value selected from the selected reference shape feature values that are close to the corresponding feature point position and the reference point position obtained from the reference direction vector. It is possible to accurately select a reference shape feature amount obtained for a reference feature point set in a posture model similar to the posture of the person and set in the same part as the part in which the feature point is set. Therefore, this image monitoring apparatus can accurately estimate the posture of the person shown in the monitoring image.
In addition, since this image monitoring device uses the reference shape feature amount in units of parts such as limbs, if attention is paid to a specific part, there is only one posture model in which the angles of the joints included in the part are all the same. Good. That is, it is not necessary to prepare a posture model in which the joint angles are changed little by little for each of the plurality of parts. Therefore, since the number of posture models that must be prepared can be reduced, the number of reference shape feature amounts can be reduced. As a result, the image monitoring apparatus can reduce the amount of calculation required for calculating the similarity between the feature amounts, and thus the amount of calculation for the entire posture estimation can also be reduced.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the posture estimation unit 77 may include a discriminator that receives a parameter representing the joint angle of each part and outputs an identification number representing the posture. Such a discriminator can be constituted by, for example, a perceptron type neural network. In this case, the discriminator is pre-learned using, for example, back propagation, using a parameter representing the joint angle of each part obtained from a plurality of posture models whose postures are known in advance. Therefore, the posture estimation unit 77 can obtain an appropriate posture estimation result by inputting a parameter representing the joint angle of each part estimated by the part shape estimation unit 76 to the discriminator. Similarly, the determination unit 78 may include a discriminator that receives a parameter indicating the angle of the joint of each part and outputs a value indicating whether it is normal or abnormal. Also in this case, the discriminator can be constituted by, for example, a perceptron type neural network. The discriminator is pre-learned using, for example, back propagation, using a parameter representing the angle of the joint of each part obtained from a plurality of posture models that are known in advance as normal or abnormal.

  Further, depending on the accuracy required for posture estimation, the processing of the whole body reference point specifying unit may be omitted. In this case, the image monitoring apparatus estimates the posture of each part by directly performing the process of the partial shape estimation unit using the reference shape feature value selected by the feature value selection unit.

Furthermore, as in the case where the monitoring area is set in front of the automatic teller machine, the imaging unit 2 may capture a part of the person whose posture is to be estimated, for example, only the upper body. In this case, for example, the posture estimation device 3 estimates the angles of the joints of only the right arm, only the left arm, or both arms, and estimates the posture of the upper body of the person according to the joint angles of those portions.
As described above, those skilled in the art can make various modifications within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Image monitoring apparatus 2 Imaging part 3 Posture estimation apparatus 4 Interface part 5 Memory | storage part 6 Output part 7 Control part 71 Feature point setting part 72 Partial shape feature-value calculation part 73 Similarity degree calculation part 74 Feature-value selection part 75 Whole body reference point specification Unit 76 region shape estimation unit 77 posture estimation unit 78 determination unit

Claims (3)

A posture estimation device for estimating the posture of a person in an image,
A feature point setting unit for detecting an edge in the image and setting a plurality of feature points on the edge;
For each of the plurality of feature points, a feature amount calculation unit that calculates a shape feature amount indicating a distribution of positional relationships with other plurality of feature points;
For a plurality of reference feature points set in each of a plurality of posture models representing a posture of a person, a reference shape feature amount that is the shape feature amount obtained in advance and specification of a posture model corresponding to the reference shape feature amount A storage unit for storing a part parameter value representing a joint angle for each part;
A feature quantity selection unit that selects, for each feature point, the reference shape feature quantity similar to each of the shape feature quantities;
A distribution for each specific part is obtained for the part parameter value corresponding to the selected reference shape feature amount, and the specific part of the person shown in the image from the part parameter value where the distribution of the part parameter value is concentrated A part shape estimation unit for estimating the shape of
A posture estimation unit that estimates a posture of a person shown in the image using the estimated part shape;
A posture estimation device having The storage unit stores an index representing a direction and a distance from the reference feature point to a reference standard point for representing the position of the whole body of the posture model in association with the reference shape feature amount;
For each reference shape feature value selected by the feature value selection unit, the index associated with the reference shape feature value and the position of the feature point corresponding to the shape feature value similar to the reference shape feature value A whole body criterion for obtaining a voting position for a reference point representing the position of a person on the image from the image and selecting the reference shape feature quantity having the voting position within a predetermined range from the most frequent position where the voting position is most concentrated It further has a point identification part,
The posture according to claim 1, wherein the partial shape estimation unit uses only the reference shape feature amount selected by the whole body reference point specifying unit among the reference shape feature amounts for estimation of the part shape of the person. Estimating device. The storage unit further stores the degree of weight of the posture model in association with the reference shape feature amount,
The whole body reference point specifying unit is configured to determine the position of the reference point on the image obtained by using the reference shape feature value selected by the feature value selection unit, and the fat associated with the reference shape feature value. The posture estimation apparatus according to claim 2, wherein the voting position is determined as a multidimensional value including a degree.