JP7015152B2 - Processing equipment, methods and programs related to key point data - Google Patents

Description

The present invention is a processing device for keypoint data capable of obtaining information related to a predetermined posture in a three-dimensional space even when keypoint data such as skeletal joint data is given as two-dimensional data. , Methods and programs.

In recent years, with the spread of motion capture sensors, especially the inexpensive sensor Kinect (registered trademark), 3D skeletal joint data (data giving skeletal joint positions in a predetermined 3D space, as shown in Fig. 1). It has become easier to acquire skeleton data). Taking the opportunity of facilitating the acquisition of the various 3D data, for example, a deep layer targeting gestures, facial expressions, etc. expressed by the 3D data, as disclosed in Non-Patent Document 2. A recognition technique using learning has been proposed.

On the other hand, in recent years, techniques for estimating whole-body or facial skeletal joint data based only on ordinary images have also matured. This technology does not require special sensors including Kinect, and has the advantage that only ordinary cameras such as familiar Web cameras need to be used, but at present, skeletal joint data etc. are used as 3D data from images. It is difficult to estimate, and it is limited to estimating skeletal joint data as two-dimensional data of coordinates on the image.

As the two-dimensional estimation technique, for example, Non-Patent Document 1 realizes estimation of two-dimensional skeletal joint data of a plurality of people at the same time from an image, and simultaneously calculates the reliability from 0 to 1 for each joint.

Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR2017 (2017). Ke, Qiuhong, et al. "A New Representation of Skeleton Sequences for 3D Action Recognition." CVPR2017 (2017). Wu, Ren, et al. "Deep image: Scaling up image recognition." ArXiv preprint arXiv: 1501.02876 7.8 (2015). Wang, Limin, et al. "Temporal segment networks: Towards good practices for deep action recognition." European Conference on Computer Vision. Springer International Publishing, 2016. Shen, Jie, et al. "The first facial landmark tracking in-the-wild challenge: Benchmark and results." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2015. Simon, Tomas, et al. "Hand Keypoint Detection in Single Images using Multiview Bootstrapping." ArXiv preprint arXiv: 1704.07809 (2017).

However, in Non-Patent Document 1 and the like as described above, it is necessary to perform recognition processing and the like for 2D skeletal joint data and the like, which cannot obtain 3D information in principle because they can be obtained only from ordinary images. When considering it, it is desirable to correct it to a predetermined posture such as the front surface as pretreatment, that is, to process it, but the realization of such processing was not considered as an issue in any of the prior arts. However, such processing has not been feasible with the prior art.

That is, such processing is extremely challenging due to the principle restrictions of lack of depth information or three-dimensional information and lack of semantic information, and cannot be realized by conventional technology. Further, the processing is expected to contribute to the augmentation of learning data in the recognition processing.

In addition, as the technology related to the problem, there are the following processing processes in consideration of recognition processing and the like, but none of them could realize the above-mentioned processing.

For example, in Non-Patent Document 3 for general image recognition, color casting is used to bias the entire image to a specific color such as red or blue as a processing process for enhancing data used in deep learning. Processing, peripheral dimming (vignetting) processing that makes the peripheral part of the image darker than the central part, lens distortion as a conversion that expands the rectangular area in the central part of the image into a barrel shape or conversely contracts it into a thread winding type ( Although lens distortion) processing is mentioned, these cannot realize the processing to correct to the predetermined posture such as the front surface.

Further, in Non-Patent Document 4, which aims to further apply deep learning succeeded in still images to behavior recognition in images, the images are divided into a plurality of fragments (snippets) on the time axis, and then the plurality of fragments are used. It proposes to learn a deep network, and cites rectangular cropping (corner cropping) processing and scale jittering (scale jittering) processing as processing processes for data enhancement during the learning. In rectangular trimming, a 1/4 size image is trimmed by dividing the image area into 2 × 2 = 4 equal parts, and the 1/4 size image is also trimmed from the center part. In scale jittering, the vertical and horizontal sizes of the cropped image are randomly determined from the resolution candidates {256, 224,192,168}, and then resized to a predetermined size of 224x224. The data enhancement by the rectangular trimming and scale jittering in Non-Patent Document 4 basically follows the method for still images targeted by Non-Patent Document 3 and the like, and also moves to the predetermined posture such as the front surface. It is not possible to realize the processing to correct.

In view of the above problems of the prior art, the present invention can obtain information related to a predetermined posture in a three-dimensional space even when key point data such as skeletal joint data is given as two-dimensional data. The purpose is to provide processing equipment, methods and programs related to key point data.

In order to achieve the above object, the present invention is a processing device for keypoint data, and is a candidate for obtaining a plurality of processing data by applying a geometric transformation to each of the two-dimensional keypoint data extracted from the whole body or a part thereof. The most similar data between the generation unit, the plurality of reference two-dimensional keypoint data obtained by two-dimensionally mapping the plurality of three-dimensional keypoint data in a predetermined posture, and the plurality of obtained machining data. It is characterized by comprising a search unit for searching for the most similar machining data and / or outputting information on the geometric transformation applied to the machining data. Further, it is characterized in that it is a method and a program corresponding to the processing apparatus.

According to the present invention, after applying geometric transformation to each of the two-dimensional keypoint data in the candidate generation unit to obtain a plurality of processing data, the plurality of three-dimensional keypoint data are two-dimensionally mapped in a predetermined posture. Information on the most similar machining data and / or geometric transformation applied to the machining data by searching the search unit for the most similar ones among the obtained multiple reference two-dimensional key point data. As a result, it is possible to obtain information related to a predetermined posture in a three-dimensional space.

It is a figure which shows the schematic example of 3D skeletal joint data. It is a functional block diagram of the processing apparatus which concerns on one Embodiment. It schematically shows the data format in MSRAction3D as an example of a 3D skeleton data set. It is a figure which shows the definition example of the predetermined polygon which is the object of area calculation for frontal determination when the data format of FIG. 3 is used. As an example of the key point data to which the present invention can be applied in addition to the whole body skeletal joint data, it is a figure which shows the schematic example of the skeletal joint data of a face and the skeletal joint data of a hand.

FIG. 2 is a functional block diagram of the processing apparatus according to the embodiment. As shown in the figure, the processing apparatus 10 includes a candidate generation unit 1, a reference target generation unit 2, and a search unit 3, and the schematic functions of each functional unit are as follows. In FIG. 2, in addition to the functional blocks, illustrations expressing schematic examples of the data processed in each part are drawn in parentheses from the viewpoint of promoting understanding of the explanation.

The candidate generation unit 1 receives the two-dimensional key point data D0 as an input, and applies various N types of processing Pi (i = 1, 2, ..., N) to the two-dimensional key point data Di (i). = 1,2, ..., N) are obtained respectively, and each of the plurality of machining data Dis is output to the search unit 3. The reference target generation unit 2 receives multiple (M) 3D key point data Rj (j = 1,2, ..., M) for reference prepared as a database constructed in advance as input. , By applying conversion processing etc. to this, the normalized 2D keypoint data Qj (j = 1,2, ..., M) in the predetermined posture is generated, and the generated 2D keypoint data Qj is generated. Is output to the search unit 3 respectively.

In the search unit 3, a plurality of two-dimensional key point data Di obtained by processing in the candidate generation unit 1 and a plurality of reference two-dimensional key point data Qj generated in the reference target generation unit 2 are used. Search for the most similar ones between. For the sake of explanation, assuming that the data Di and Qj determined to be the most similar by the search are represented as the data Di _max and Qj _max specified by i = i _max and j = j _max , respectively, the search unit 3 further The machining data Di _max determined to be the most similar is output as a machining result for the original data D0 input to the candidate generation unit 1. The output from the search unit 3 may be replaced with or in addition to the machining result data Di _max , and may be the machining content Pi _max for the original data D0.

The similarity score of the data Di and the data Qj calculated by the search unit 3 for the search (the score obtained by quantifying the similarity so that the higher the similarity is, the larger the value is) is called score (Di, Qj). Expressed, i = i _max and j = j _max corresponding to the most similar results can be written as (1) below.

Hereinafter, the details of the processing contents of each functional unit in FIG. 2 will be described in the order of the reference target generation unit 2, the candidate generation unit 1, and the search unit 3. Here, a case where the key point data to be processed as two-dimensional or three-dimensional data in each functional unit is specifically skeletal joint data will be described as an example, but the key point data is of a type other than skeletal joint data. It may be a thing. Similarly, the illustration as a schematic example of the data Di, data Qj, etc. in FIG. 2 is drawn as an example when the key point data is skeletal joint data, but this is for clarification of the explanation. The key point data may be of a type other than the skeletal joint data, which is merely an example. Key point data of types other than skeletal joint data will be described later.

<Reference target generator 2>
In the reference target generation unit 2, the two-dimensional skeletal joint data of the frontal posture is generated from each data Rj of the existing three-dimensional skeletal joint data set prepared in advance as a model of the whole body skeleton in a predetermined format. Data Qj is obtained by normalizing the generated 2D skeletal joint data. As the 3D skeleton data set of the predetermined format, for example, the MSR Action 3D format used as a data set for evaluation in the academic / technical fields of behavior recognition can be used.

FIG. 3 schematically shows the data format in MSRAction3D as an example of the three-dimensional skeleton data set. (Note that FIG. 1 above is also a schematic example of the data format.) As shown in FIG. 3, the data of MSRAction3D is the predefined 20 joints and 19 bones (connection relationship between joints). ) Is used to represent the whole body skeleton as a model, and the posture of a character such as a person can be expressed by giving the three-dimensional coordinates of each of the 20 joints. In addition, by giving the data in time series, it is possible to express the behavior such as the gesture of the character. Since the character of the three-dimensional skeleton data in FIG. 3 represents the state in which the face faces the paper, the joints on the right and left sides of the paper correspond to the joints on the left and right sides of the character. ..

As shown in FIG. 3, the MSRAction3D data defines joints and their connection relationships as follows. That is, "head-neck-spine-center hip" is defined in the sitting height part, and "neck-right shoulder-right" in the right arm side part. "Right elbow-right wrist-right hand" is defined, the same is defined as the left side of the left arm side, and the right leg side is "center hip-buttock". "Right hip-right knee-right ankle-right foot" is defined, and the same is defined as the left side of the left leg side.

The 3D skeleton data set read by the reference target generation unit 2 is not limited to the MSRAction3D format, and similarly, any predetermined format defined by modeling the joints and their connection relationships is used. be able to.

Specifically, the reference target generation unit 2 can generate each data Qj in the two-dimensional format from each data Rj in the three-dimensional format by the following procedures 21, 22, 23.

(Procedure 21)
As expressed by the following equations (2-1) and (2-2), fluoroscopic projection conversion (that is, 3D) is performed under various preset parameter candidates θ _x , θ _y , and θ _z , respectively. 3D data Rj is mapped to 2D data Bj (θ _x , θ _y , θ _z ) by performing a transformation known as mathematics dealing with CG (computer graphics) and the like.

Here, (a _x , a _y , a _z ) is one 3D joint coordinate in the 3D skeletal joint data, and (c _x , c _y , c _z ) is a 3D preset. It is the camera coordinates, and (θ _x , θ _y , θ _z ) as each parameter candidate is the direction of the camera for two-dimensional projection (Tait-Bryan angles of Euler angles), and is expressed in Eq. (2-1). The (b _x , by) obtained by Eq. (2-2) via the calculated (d _x , d _y , _{d z} ) is the 2D of the 3D skeletal coordinates (a _x , a _y , a _z ). It becomes the coordinates as a projection result.

That is, each point (a _x , a _y , a _z ) ∈ Rj constituting the three-dimensional data Rj is set to the parameter (θ _x , θ _y , θ _z ) according to the above (2-1) and (2-2). Under, it can be converted into each point (b _x , by) _∈ Bj (θ _x , θ _y , θ _z ) of the two-dimensional data.

The conversion according to the equations (2-1) and (2-2) is that the camera position coordinates (c _x , c _y , c _z ) are fixed and the camera angle (θ _x , θ _y , θ _z ) is used as a parameter. However, it is not limited to this, and if mapping is performed using arbitrary parameter settings and conversion relational expressions that express the 3D skeletal joint data Rj as 2D data when viewed from various viewpoints. good. That is, the 3D data Rj is mapped as data on the image plane obtained by shooting from various viewpoints specified by the parameters using a predetermined camera (a pinhole camera as a theoretical model in which blurring does not exist). It is only necessary to obtain two-dimensional data.

For example, the camera angle (θ _x , θ _y , θ _z ) is set from the camera position toward a predetermined point (for example, the center of gravity) of the 3D data Rj, and a predetermined spherical surface (for example, the center of the center) that covers the 3D data Rj is set. The camera position coordinates (c _x , _{cy, c z )} sampled uniformly from above (a spherical surface of a predetermined size) may be used as a parameter. However, from the viewpoint of properly implementing the following procedure 22, it is desirable to set parameters so that the distance when viewing the 3D skeletal joint data Rj from various viewpoints does not change significantly.

In the following explanation, when explaining the positional relationship in the two-dimensional data, from the viewpoint of promoting an intuitive understanding of the explanation, the one corresponding to the coordinates of the two-dimensional data is appropriately referred to as an "image" or the like. In addition, terms related to "image" (number of pixels, aspect ratio, etc.) will be appropriately used for explanation.

(Procedure 22)
From the two-dimensional data Bj (θ _x , θ _y , θ _z ) obtained for each parameter (θ _x , θ _y , θ _z ) in step 21, the three-dimensional data Rj is obtained from the front as an example of a predetermined posture. Determine the data Bj (θ _x , θ _y , θ _z ) _[front] that corresponds to what you see.

Specifically, the area of the polygon formed by tracing the predetermined joints in the two-dimensional data Bj (θ _x , θ _y , θ _z ) in a predetermined order is the area (Bj (θ _x , θ _y , θ _z )). )), The parameters (θ _x , θ _y , θ _z ) that give the maximum value of the area can be determined as the parameters corresponding to the front surface. Expressed in the formula, it is as follows (2-3).

Here, regarding the predetermined joint for forming the polygon to be calculated for the above area area (Bj (θ _x , θ _y , θ _z )) and the predetermined order to follow the joint, the reference target generation unit 2 is used. A predetermined one may be defined corresponding to a predetermined type of joint defined in the read 3D skeletal joint data. In making the definition, it may be defined as satisfying the following first and second properties.
(First property) When the character corresponds to the front regardless of the pose type of the character (the pose type corresponding to the data Rj, which may include the distinction between the characters themselves (adults and children, etc.). The same shall apply hereinafter). The area of the polygon formed in is maximized. (It should be noted that it is not always necessary to be strictly in front and to be strictly maximum, and it is sufficient that the area tends to increase as it is closer to a certain direction that can be regarded as the front by manual judgment or the like.)
(Second property) Even if the pose types of the characters are different and their sizes are different, the relative shape of the polygon (when it corresponds to the front seen in the three-dimensional space). The shape) does not differ significantly.

Specifically, in the case of the MSRAction3D data format described in FIG. 3, for example, a polygon (polygon indicated by a thick line) such as [1] or [2] in FIG. 4 is defined for area calculation. Can be kept. [1] in FIG. 4 is an example of defining a pentagon in which each vertex (that is, each joint) is formed side by side in the order of "head → left shoulder → left buttock → right buttock → right shoulder → head" as an area calculation target. Is. [2] in FIG. 4 is an example of defining a quadrangle formed by arranging each vertex (that is, each joint) in the order of "neck-> left shoulder-> center of buttocks-> right shoulder-> neck" as an area calculation target.

In each of the examples [1] and [2] of FIG. 4, a polygon is defined from the joint of the part generally corresponding to the body of the character, and the above-mentioned first and second characteristics are obtained by having the following first and second characteristics. It satisfies the first and second properties.
(First characteristic) Even if the pose type is different, the body shape itself does not change so much (except for the size change when the pose character is different).
(Second characteristic) Moreover, the unchanged fuselage shape is a generally planar shape, and all the coordinates of the fuselage-related joints in the three-dimensional space are virtual planes penetrating the fuselage (plane corresponding to the front direction). It is within a certain distance from, and is generally in a coplanar relationship.

It should be noted that, for example, if a polygon is tentatively defined including the joints of the arms and legs, which may be accompanied by a large relative position change due to the difference in pose type, instead of the joints related to the torso, it has the above-mentioned first and second characteristics. Even if it does not exist, it does not satisfy the above-mentioned first and second properties.

(Procedure 23)
By normalizing the frontal data Bj (θ _x , θ _y , θ _z ) _[frontal] obtained in step 22 above, the original 3D data Rj is 2D mapped and normalized in a predetermined posture. Get the data Qj. The normalization is to enable the search unit 3 to appropriately search for similar data. Specifically, the normalization regarding the size (scale) and the position may be performed as follows.

That is, as the normalization of the size, the distance dj between the predetermined two points among the position coordinates of the skeletal joint in the frontal data Bj (θ _x , θ _y , θ _z ) _[frontal] is a predetermined fixation for normalization. The size of the front data is multiplied vertically and horizontally (that is, without changing the aspect ratio) d _{[normalization]} / dj so that the value d _{[normalization]} (for example, 50 pixel value) is obtained. Here, the predetermined two points that give the distance dj may be set so that the length between the two points is stable and does not change regardless of the pose type in the frontalized skeletal joint data. .. For example, in the case of the data format of FIG. 3, the two points are "head and neck", "neck and spine" or "spine and center of the hip (spine)". You can set one of "center hip)". It should be noted that these examples are related to the central part of the fuselage when faced in step 22, and it is assumed that the distance between the two points is stable.

In addition, as a normalization of the position, translational movement is performed so that a predetermined point is the origin of the position coordinates of the skeletal joint in the frontal data Bj (θ _x , θ _y , θ _z ) _[frontal] . The predetermined point as the origin by the translational movement may be, for example, one of the two points used for the normalization of the size, for example, two points of "head" and "neck". When normalized by the distance, the coordinates of the neck should be set to be the origin. Assuming that the coordinates of the predetermined point as the origin are (x _[reference] , y _[reference] ), the position coordinates of each skeletal joint in the frontal data Bj (θ _x , θ _y , θ _z ) _[frontal] are before normalization. From (x _{[before normalization]} , y _{[before normalization]} ), it will be normalized as shown on the left side of the following equation (2-4).

Since the above normalization of the size and position gives the same result regardless of the order, the normalization may be performed in any order. In addition, since candidates rotated at various angles are generated in the candidate generation unit 1 described below, the above-mentioned size and position are normalized in the procedure 23 of the reference target generation unit 2, although the above-mentioned size and position are normalized. There is no need to perform axial normalization.

<Candidate generation unit 1>
The candidate generation unit 1 receives the two-dimensional skeletal joint data D0, and applies a plurality of predetermined processing Pis (i = 1, 2, ..., N) as various geometric transformations to the two-dimensional skeletal joint data D0. Obtain the machining data Di (i = 1,2, ..., N) respectively. The machining process Pi as the geometric transformation to be performed is predetermined as a plurality of processes that can at least approximately correct the two-dimensional data D0 to the data in the case of the front as a predetermined posture by one of the machining processes. You can set the parameter range. The policy for setting the parameter range is almost the same as the policy for setting the parameters for the perspective projection conversion in step 21 in the reference target generation unit 2, that is, the viewing angle covers all directions on the spherical surface. Can be adopted. It is desirable to set multiple geometric transformations so as to cover the difference in appearance due to the difference in viewing angle.

Specifically, for example, the processing Pi is set to include a transformation or rotation transformation for frontalization, and is given a parameter for specifying the details of the transformation in the affine transformation and / or the homography transformation. be able to. As is well known, the affine transformation corresponds to a special case of the homography transformation. Specifically, in the candidate generation unit 1, the data Di obtained by processing the data D0 by the processing Pi can be obtained by the following procedures 11 and 12.

(Procedure 11)
Data Ei∋ (x _{[conversion] by performing affine transformation or homography transformation (conversion specified by a predetermined parameter corresponding to the processing Pi) for data D0∋ (x [before conversion] ,} y _{[before conversion]} ) _After] , y _{[after conversion]} ) is obtained. As is well known as a mathematical relationship, the affine transformation (matrix element a _ij ) and homography transformation (matrix element h _ij ) in two dimensions using homogeneous coordinate representation are performed by the following equations (3-1) and (3), respectively. It can be expressed as -2).

(Procedure 12)
The processed data Di is obtained by normalizing the data Ei obtained in step 11 above. Since the normalization may be performed by applying the same processing as the normalization already described in step 23 of the reference target generation unit 2 to the data Ei, duplicate explanations will be omitted.

The two-dimensional skeletal joint data D0 read by the candidate generation unit 1 can be prepared by extracting the data D0 by, for example, the method of Non-Patent Document 1 described above. In Non-Patent Document 1, data D0 is calculated by deep learning for both maps after extracting a part confidence map for the body part and a part affinity vector map for the body part from the captured image of the character. At this time, the reliability of each joint constituting the data D0 is also calculated. Here, in order to realize the processing of the search unit 3 described next with high accuracy, it is preferable to associate the reliability with each joint of the data D0. Further, it is assumed that the two-dimensional data D0 read by the candidate generation unit 1 is defined for the same type of joint as defined in the data format of the three-dimensional data Rj read by the reference target generation unit 2. For example, if the 3D data Rj is in the MSRAction3D data format as described in FIG. 3, the 2D data D0 is also prepared to give the 2D coordinates of each joint as shown in FIG. 3 defined in the data format. I will try to do it.

<Search unit 3>
The search unit 3 is obtained from the processing data Di (two-dimensional skeletal joint data) for each processing Pi (i = 1, 2, ..., N) obtained from the candidate generation unit 1 and the reference target generation unit 2. The pair that is the most similar between the data Qj in which multiple 3D skeletal joint data Rj (j = 1,2, ..., M) are faced as predetermined postures and expressed as 2D skeletal joint data. The data Di _max and Qj _max are searched for, and the machining data Di _max and / or the machining content Pi _max in the search result is output as an output from the machining device 10. The search unit 3 can specifically perform the search process according to the following procedures 31 and 32.

(Procedure 31)
For each machining data Di, the most similar data Qj _{max [i]} is obtained by calculating the similarity score score (Di, Qj) with the data Qj (j = 1,2, ..., M). Search and record the similarity score score [i] = score (Di, Qj _{max [i]} ). Regarding the similarity score score (Di, Qj), the distance L (Di, Qj) between the two data is calculated as shown in (4), (5), (6) below, and the smaller the distance, the smaller the distance. It may be calculated as score (Di, Qj) = f (L (Di, Qj)) using a predetermined decreasing function f so that the similarity score becomes large.

In equation (4) for calculating the distance, x (Di) _k is the two-dimensional position coordinate of the k-th predetermined joint of the data Di, and x (Qj) _k is the two-dimensional position of the k-th predetermined joint of the data Qj. As mentioned above, the data Di and Qj (the original Rj) are the coordinates, and the same kind of data is adopted, and the types of these k-th predetermined joints are the same. For example, in the case of using the data format of MSRAction3D in FIG. 3, if the k = 1st joint of Di is the “head”, the k = 1st joint of Qj is also the “head”. K is the total number of joints defined in the Di and Qj data formats.

Then, with respect to the calculation of the weight w _k for calculating the distance L (Di, Qj) of the equation (4), the equations (5) and / or (6) can be used, and c _k is the kth of Di. The reliability calculated by the method of Non-Patent Document 1 or the like described above for the joint (the higher the probability of corresponding to the joint, the higher the reliability calculated with a non-negative value). TH1 and TH2 in Eq. (5) are threshold values for the reliability. If the reliability is low, the weight value is set to low 0, if the reliability is medium, the weight value is set to medium 0.5, and if the reliability is high, the weight value is set to 0.5. Is set to high 1.

When both equations (5) and (6) are used, the right-hand side w _k calculated in equation (6) should be read as "normalized reliability c _{k [normalization]} ". The "normalized reliability c _{k [normalization]} " may be used for the threshold determination in Eq. (5), and the weight value may be calculated to be 0, 0.5, or 1.

If the reliability is not linked to each joint of the data Di (when the reliability information is not available), set all the weights as equal values such as w _k = 1, and formula (4). ) To calculate the distance.

Further, as another embodiment of the equations (4), (5), and (6), the reliability c _k is calculated as a non-negative value, and the reliability c _k remains as it is by setting w _k = c _k . The distance may be calculated from Eq. (4) by adopting it as the weight w _k . In addition, Eq. (5) was an example in which the reliability value was divided into three stages by threshold values and weights were given according to their magnitudes. Similarly, the reliability values were divided into any number of stages and weights were given according to each stage. You may. Similarly, as a modification of Eq. (5), the weight may be calculated as w _k = g (c _k ) using a predetermined increasing function g (which takes a non-negative value).

(Procedure 32)
Of the most similar score score [i] = score (Di, Qj _{max [i]} ) obtained for each data Di by the above procedure 31, the one corresponding to the maximum value is determined as the most similar result Di _max and corresponds. Information (machining information Pi _max and / or machining data Di _max ) is output as a result.

As described above, in the present invention, the two-dimensional skeletal joint data D0 is processed into various candidate Dis in the candidate generation unit 1, and the data Qj as the frontalized data generated by the reference target generation unit 2 from a large number of three-dimensional data Rj. By comparing with the search unit 3, the problem of shooting angle dependence can be solved. That is, it is possible to solve the shooting angle dependence problem when the data D0 is obtained by extracting from the image by the method as in Non-Patent Document 1. Here, compared with the image data, the skeletal joint data has higher-order information such as joint information and is reflected in the calculation of the equation (4), so that the shooting angle can be corrected more accurately. Here, further, by using the reliability of the joint position, the shooting angle can be corrected more accurately.

Hereinafter, explanatory supplements in the present invention will be described.

(1) In the above description, whole body skeletal joint data is used for explanation as an example of key point data, but the present invention is not limited to whole body skeleton data, but key points extracted from any whole body or a part thereof. It is equally applicable to data. For example, the present invention can be applied to facial skeletal joint data as shown in FIG. 5 [1A] and hand skeletal joint data as shown in FIG. 6 [1B]. Is. A key point may be a point suitable for modeling the movement or gesture of the character's whole body or part of it by tracking the key point, and may be an actual joint, a modeled joint, or other point ( It does not have to be a joint) as long as it is defined as. In the case of joints, the equivalent of "bone" between joints may or may not be defined.

The facial skeletal joint data of FIG. 5 [1A] includes the left and right eyebrows, the left and right eye circumferences, the nasal muscles and the lower part of the nose, the outer and inner circumferences of the lips, and the chin from the left and right ears through the cheeks. The joints are modeled with respect to the borders of the face leading up to. In this case, for example, the triangle formed by the three joints of the left and right outer corners of the eyes and the tip of the nose shown in black in [2A] is the area area (Bj (θ)) for frontalization in step 22 of the reference target generator 2. It may be set as the one to calculate _x , θ _y , θ _z )). For the process of obtaining the skeletal joint data of the face as the input data to the candidate generation unit 1 from the image as the two-dimensional data, for example, the process disclosed in Non-Patent Document 5 described above may be used. As for the three-dimensional data, the one defined for the same joint may be prepared and input to the reference target generation unit 2.

The skeletal joint data of the hand in FIG. 5 [1B] is modeled by defining bones between the wrist joints and the third, second, first and fingertip joints of each of the first to fifth fingers. It was done. In this case, for example, refer to the hexagon defined by the wrist joint and the third joint of each finger shown in black in [ _2B ]. , θ _y , θ _z ))) may be set to be calculated. The process of obtaining the skeletal joint data of the hand as the input data to the candidate generation unit 1 from the image as the two-dimensional data may be, for example, the process disclosed in Non-Patent Document 6 described above. As for the three-dimensional data, the one defined for the same joint may be prepared and input to the reference target generation unit 2.

When using facial skeletal joint data, the present invention can be used, for example, as preprocessing for facial expression recognition, and when using hand skeletal joint data, the present invention is, for example, for recognizing hand operation contents and work contents. It can be used as pretreatment.

(2) The present invention has described the case where the search unit 3 obtains the result corresponding to the front direction as an example of the predetermined posture, but the search unit 3 obtains the result corresponding to the predetermined posture in the tilted direction with respect to the front. You can also do it. In this case, after grasping the front direction of the data Rj in the space by obtaining the front parameters (θ _x , θ _y , θ _z ) _[front] in step 22 of the reference target generation unit 2, the equation (2-). By applying the perspective projection conversion of 1) and (2-2) to the predetermined direction tilted from the front direction, the data Bj (θ) of the tilted predetermined posture that is the target of the normalization processing according to the following procedure 23. _x , θ _y , θ _z ) It suffices to prepare a _{[tilted predetermined posture]} .

(3) If the front direction is defined in advance in the 3D data Rj, the parameter search in step 22 in the reference target generation unit 2 is omitted, and in step 21, the perspective projection conversion is performed only in the defined front direction. By doing so, it is sufficient to prepare the two-dimensional data to be normalized by the procedure 23.

(4) By preparing the two-dimensionally mapped data Qj in advance, the process of obtaining Qj from the data Rj in the reference target generation unit 2 may be omitted.

(5) The processing apparatus 10 can be realized as a computer having a general configuration. That is, a CPU (central processing unit), a main storage device that provides a work area for the CPU, an auxiliary storage device that can be configured with a hard disk, SSD, etc., an input interface that receives input from users such as a keyboard, mouse, touch panel, etc., and a network. The processing unit 10 can be configured by a general computer including a communication interface for connecting to and communicating with, a display for displaying, a camera, and a bus connecting them. Further, the processing of each part of the processing apparatus 10 shown in FIG. 2 can be realized by a CPU that reads and executes a program for executing the processing, but any part of the processing can be performed by a separate dedicated circuit or the like (GPU). It may be realized in).

10 ... Processing equipment, 1 ... Candidate generation unit, 2 ... Reference target generation unit, 3 ... Search unit

Claims (10)

A candidate generation unit that obtains multiple machining data by applying geometric transformation to each of the two-dimensional key point data extracted from the whole body or a part thereof.
Searching for the most similar data between the plurality of reference two-dimensional keypoint data obtained by two-dimensionally mapping a plurality of three-dimensional keypoint data in a predetermined posture and the plurality of obtained machining data. It comprises a search unit that outputs the most similar machining data and / or the information of the geometric transformation applied to the machining data.
A reference target generation unit for obtaining the plurality of reference two-dimensional keypoint data by two-dimensionally mapping the plurality of three-dimensional keypoint data in a predetermined posture is further provided.
The reference target generation unit performs fluoroscopic projection conversion specified by each of the plurality of parameters for each of the plurality of three-dimensional key point data to obtain a plurality of candidates for the reference two-dimensional key point data, and the plurality of candidates. A processing device for keypoint data, which is characterized by obtaining two-dimensional keypoint data for reference based on the one having the maximum area of a polygon formed with a plurality of predetermined keypoint data as vertices. .. The reference target generation unit specifies the parameters of the perspective projection conversion in the polygon having the maximum area as those corresponding to the front surface, and determines the predetermined posture for two-dimensional mapping based on the specified front surface. The processing apparatus for the key point data according to claim 1 , wherein the processing apparatus relates to the key point data. The processing apparatus for key point data according to claim 2 , wherein the predetermined posture is in the specified front surface or is tilted from the front surface. The present invention according to any one of claims 1 to 3 , wherein the candidate generation unit applies the affine transformation or homography transformation specified by a predetermined parameter as the geometric transformation to obtain the plurality of processing data, respectively. Processing equipment for key point data of. In the candidate generation unit, a geometric transformation is applied to further set a predetermined point in the two-dimensional keypoint data as a coordinate reference point, and the length between the predetermined two points in the two-dimensional keypoint data is used as a reference. The processing apparatus for key point data according to any one of claims 1 to 4 , wherein the plurality of processing data are obtained as scale-converted to a value. For each of the plurality of reference two-dimensional key point data, a predetermined point in the two-dimensional key point data is set as a coordinate reference point, and the length between the predetermined two points in the two-dimensional key point data is a reference value. The processing apparatus for key point data according to any one of claims 1 to 5 , wherein the data is obtained as scale-converted so as to be. Reliability is associated with each point of the two-dimensional key point data that is the source of obtaining a plurality of machining data in the candidate generation unit.
The processing apparatus for key point data according to any one of claims 1 to 6 , wherein when the search unit searches for the most similar one, the linked reliability is taken into consideration. Each of the two-dimensional and three-dimensional key point data is characterized by being modeled as either whole body skeletal joint data, facial skeletal joint data capable of reflecting facial expressions, or hand skeletal joint data. The processing apparatus related to the key point data according to any one of claims 1 to 7 . It is a processing method related to key point data executed by a computer.
Candidate generation stage to obtain multiple processing data by applying geometric transformation to 2D key point data extracted from the whole body or a part of it,
Searching for the most similar data between the plurality of reference two-dimensional keypoint data obtained by two-dimensionally mapping a plurality of three-dimensional keypoint data in a predetermined posture and the plurality of obtained machining data. It comprises a search step that outputs the most similar machining data and / or the information of the geometric transformation applied to the machining data.
Further provided with a reference target generation stage for obtaining the plurality of reference two-dimensional keypoint data by two-dimensionally mapping the plurality of three-dimensional keypoint data in a predetermined posture.
In the reference target generation stage, each of the plurality of three-dimensional key point data is subjected to fluoroscopic projection conversion specified by each of the plurality of parameters to obtain a plurality of candidates for the reference two-dimensional key point data, and the plurality of candidates are obtained. A processing method for keypoint data, which comprises obtaining two-dimensional keypoint data for reference based on the one having the maximum area of a polygon formed with a plurality of predetermined keypoint data as vertices. .. A program for keypoint data, characterized in that the computer functions as a processing device for the keypoint data according to any one of claims 1 to 8 .

Images