JP2018089161A - Target person recognition method, device, system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately determine a probability that a target person is a specific individual on the basis of a distance image of a target person that is to be obtained from a distance image sensor.SOLUTION: A target person recognition method is the method to be executed by a computer. The method includes: calculating a first distance corresponding to a first physical feature amount of a target person and a second distance corresponding to a different second physical feature amount of a target person on the basis of position data of a body region of a target person that is to be obtained from a first distance image with distance information of a target person as a pixel value; and determining a probability that a target person is a specific individual on the basis of a ratio of a first distance to a second distance and body shape information to be acquired from a storage section for storing body shape information related to a body shape of a specific individual.SELECTED DRAWING: Figure 3

Description

  The present disclosure relates to a subject recognition method, a subject recognition device, a subject recognition system, and a subject recognition program.

  In order to prevent misrecognition of persons other than the target person, a target person who performs a predetermined action based on a predetermined feature in the distance image and personal information representing an individual's actual height, chest circumference, abdominal circumference, etc. A technique for identifying the operation information is known.

JP 2015-61577 Japanese Patent Laid-Open No. 2003-162720 JP 2010-233737 A

  However, with the conventional techniques as described above, it is difficult to accurately determine the possibility that the subject is a specific individual based on the distance image of the subject obtained from the distance image sensor. Even if the subject is the same, length information such as the height of the subject that can be calculated based on the distance image varies depending on the distance between the distance image sensor and the subject. Therefore, it is difficult to accurately determine the possibility that the subject of the distance image is a specific individual only by using the above personal information.

  Therefore, in one aspect, an object of the present invention is to accurately determine the possibility that the subject is a specific individual based on the distance image of the subject obtained from the distance image sensor.

In one aspect, based on the skeleton information obtained from the first distance image having the distance information of the subject person as a pixel value, the first distance corresponding to the first body feature amount of the subject person and the distinction of the subject person And a second distance corresponding to the second body feature amount of
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A computer-implemented subject recognition method is provided that includes determining.

  In one aspect, according to the present invention, it is possible to accurately determine the possibility that the subject is a specific individual based on the distance image of the subject obtained from the distance image sensor.

It is a figure which shows typically schematic structure of the subject recognition system by Example 1. FIG. It is a figure which shows an example of the hardware constitutions of a subject recognition apparatus. It is a block diagram which shows an example of the function of a subject recognition apparatus. It is a figure which shows an example of how to divide each part of the body. It is an image figure of the attention point and offset point in a distance image. It is explanatory drawing of the method of identifying a body part using a decision tree for every pixel. It is explanatory drawing which shows the outline | summary of machine learning. It is an image figure of the gravity center position of a non-joint part. It is an image figure of a gravity center position, such as a joint part. It is an image figure of the recognition result of a skeleton. It is explanatory drawing of an example of the calculation method of a gravity center position. It is explanatory drawing which shows an example of physique information. It is a figure which shows the skeleton recognition result of the person from whom a physique differs. 6 is a flowchart illustrating an example of processing executed by a subject recognition device according to Embodiment 1. It is a block diagram which shows an example of the function of the subject recognition apparatus by Example 2. It is a figure which shows an example of a robot. 12 is a flowchart illustrating an example of processing executed by a subject recognition device according to a second embodiment.

  Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[Example 1]
FIG. 1 is a diagram schematically illustrating a schematic configuration of a subject recognition system 1 according to the first embodiment. FIG. 1 shows a subject S for explanation.

  The target person recognition system 1 includes a distance image sensor 21 and a target person recognition apparatus 100.

  The distance image sensor 21 acquires a distance image of the subject S. For example, the distance image sensor 21 is a three-dimensional image sensor, measures the distance by sensing the entire space, and acquires a distance image having distance information for each pixel like a digital image. The acquisition method of distance information is arbitrary. For example, the distance information acquisition method may be an active stereo method in which a specific pattern is projected onto an object, read by an image sensor, and the distance is acquired by a triangulation method from the geometric distortion of the projection pattern. Good. Further, a TOF (Time-of-Flight) method may be used in which laser light is irradiated, reflected light is read by an image sensor, and a distance is measured from the phase shift.

  The distance image sensor 21 may be installed in a manner in which the position is fixed, or may be installed in a manner in which the position is movable. A plurality of distance image sensors 21 may be installed.

  The subject recognition apparatus 100 determines the possibility that the subject S is a specific individual based on the distance image obtained from the distance image sensor 21. A specific individual is a unique person and is hereinafter referred to as a recognition target person. Determining the possibility that the target person S is a specific individual is a concept including determining whether or not the target person S is a specific individual.

  The subject recognition device 100 may be realized in the form of a computer connected to the distance image sensor 21. The connection between the subject recognition device 100 and the distance image sensor 21 may be realized by a wired communication path, a wireless communication path, or a combination thereof. For example, when the subject recognition device 100 is in the form of a server that is relatively remote to the distance image sensor 21, the subject recognition device 100 may be connected to the distance image sensor 21 via a network. . In this case, the network may include, for example, a mobile phone wireless communication network, the Internet, the World Wide Web, a VPN (virtual private network), a WAN (Wide Area Network), a wired network, or any combination thereof. On the other hand, when the subject recognition device 100 is disposed relatively close to the distance image sensor 21, the wireless communication path is based on short-range wireless communication, Bluetooth (registered trademark), Wi-Fi (Wireless Fidelity), or the like. It may be realized.

  FIG. 2 is a diagram illustrating an example of a hardware configuration of the subject recognition device 100.

  In the example illustrated in FIG. 2, the target person recognition device 100 includes a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a drive device 104, a network I / F unit 106, and an input unit 107.

  The control unit 101 is an arithmetic device that executes a program stored in the main storage unit 102 or the auxiliary storage unit 103, receives data from the input unit 107 or the storage device, calculates, processes, and outputs the data to the storage device or the like. To do.

  The main storage unit 102 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 101. It is.

  The auxiliary storage unit 103 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 104 reads the program from the recording medium 105, for example, a flexible disk, and installs it in the storage device.

  The recording medium 105 stores a predetermined program. The program stored in the recording medium 105 is installed in the subject recognition device 100 via the drive device 104. The installed predetermined program can be executed by the subject recognition apparatus 100.

  The network I / F unit 106 is an interface between a peripheral device having a communication function and a target person recognition apparatus 100 connected via a network constructed by a data transmission path such as a wired and / or wireless line.

  The input unit 107 includes a keyboard having cursor keys, numeric input, various function keys, and the like, a mouse, a slice pad, and the like. The input unit 107 may correspond to other input methods such as voice input and gestures.

  In the example illustrated in FIG. 2, various processes described below can be realized by causing the subject recognition apparatus 100 to execute a program. It is also possible to record the program on the recording medium 105 and cause the subject recognition apparatus 100 to read the recording medium 105 on which the program is recorded, thereby realizing various processes described below. Note that various types of recording media can be used as the recording medium 105. For example, a recording medium for optically, electrically or magnetically recording information such as a CD-ROM, flexible disk, magneto-optical disk, etc., a semiconductor memory for electrically recording information such as ROM, flash memory, etc. It may be. Note that the recording medium 105 does not include a carrier wave.

  FIG. 3 is a block diagram illustrating an example of functions of the target person recognition apparatus 100. FIG. 3 also shows a distance image sensor 21 that inputs a distance image to the subject recognition apparatus 100.

  In the example illustrated in FIG. 3, the target person recognition apparatus 100 includes a body part position calculation unit 110, a recognition target person determination unit 112, a feature amount calculation unit 136 (an example of a calculation unit), and a personal recognition unit 138 (determination unit). Example). The body part position calculation unit 110, the recognition target person determination unit 112, the feature amount calculation unit 136, and the individual recognition unit 138 execute one or more programs stored in the main storage unit 102 by the control unit 101 illustrated in FIG. This can be achieved. Note that some of the functions of the subject recognition apparatus 100 may be realized by a computer that can be incorporated in the distance image sensor 21.

  The body part position calculation unit 110 includes a body part identification unit 1101, a non-joint position calculation unit 1102, and a joint position calculation unit 1103.

  A distance image of the subject S is input from the distance image sensor 21 to the body part identification unit 1101. The body part identification unit 1101 identifies each part of the body of the subject S in the distance image. Identification of each part of the body can be realized based on a learning result obtained in advance by machine learning. An example of a method for identifying each part of the body will be described later with reference to FIGS.

  FIG. 4 is a diagram illustrating an example of how to divide each part of the body. In the example shown in FIG. 4, the body is divided into 29 parts a1 to a29 as an example. The body includes joint parts (for example, a17, a18, a21, a22, etc.), non-joint parts (for example, a19, a20, a23, a24, etc.), and end parts of hands and feet (a15, a16, a28, a29). ). For example, a unique label may be given to each part of the body divided in this way. Hereinafter, identifying each body part (each body part labeled) is also referred to as “label division”.

  Here, an example of a method for identifying each part of the body will be described with reference to FIGS. Here, an example will be described in which a random forest is used as the machine learning method and the difference between the distance values of the target pixel and the surrounding pixels is used as the feature amount. In the random forest, a plurality of decision trees are used, the feature amount of the input data (distance image) is calculated, and the decision tree is branched according to the result to obtain a final result. In this example, the feature quantity is calculated for each pixel of the distance image, and the decision tree is branched to finally classify which body part (label) the pixel belongs to.

  The feature amount f (I, x) is a difference in distance value between a point of interest (pixel) in the distance image and its surrounding offset points, and is expressed by the equation (1).

ここで、Ｉは距離画像、ｘは注目点の座標、ｄ_Ｉ（ｘ）は、注目点の距離、ｄ_Ｉ（ｘ＋Δ）は、オフセット点の距離である。オフセットΔは、以下のとおりである。

Here, I is the distance image, x is the coordinates of the point of interest, d _I (x) is the distance of the point of interest, and d _I (x + Δ) is the distance of the offset point. The offset Δ is as follows.

ここで、ｖは、注目点からのオフセット（正規化前）である。尚、オフセット点の位置については学習時に決まった位置を使い、距離画像センサ２１からの対象者Ｓの近さの変動に対応するために距離で正規化を行っている。即ち、対象者Ｓが近いほど距離画像が拡大されるため、オフセットΔは、注目点の距離で正規化されている。

Here, v is an offset from the attention point (before normalization). Note that the position of the offset point is a position determined at the time of learning, and is normalized by the distance in order to cope with the variation in the proximity of the subject S from the distance image sensor 21. That is, since the distance image is enlarged as the subject S is closer, the offset Δ is normalized by the distance of the point of interest.

  FIG. 5 is an image diagram of a point of interest and an offset point in a distance image. In FIG. 5, the mark “+” represents the point of interest, the mark “x” represents the offset point, and the arrow conceptually represents the offset Δ. In FIG. 5, for example, when the attention point is a head pixel and the offset point is a background portion, the difference in the distance between the attention point and the offset point becomes large. On the other hand, when the attention point and the offset point are both in the upper body as in the body part, the difference in distance is small.

FIG. 6 is an explanatory diagram of a method for identifying a body part using a decision tree for each pixel. FIG. 6 shows T tree “tree 1” to “tree T”. The body part identification unit 1101 inputs each pixel from the distance image to a classifier (a plurality of decision trees) learned in advance, and estimates the body part for each pixel. The body part identification unit 1101 calculates the feature value f (I, x; Δ _k ) of the pixel at each node N _k of the decision tree and compares it with the threshold θ _k . Offset delta _k and the threshold theta _k to be used for calculation in each node the most correct site can identify the value at the time of learning is selected. The body part identification unit 1101 branches in the direction corresponding to the comparison result at each node, proceeds to the next node, and reaches the terminal node. When the end node of the decision tree is reached, a probability distribution as to which body part the pixel belongs to is obtained. In FIG. 6, P ₁ (c) schematically represents a probability distribution in the decision tree “tree 1”, and P _T (c) schematically represents a probability distribution in the decision tree “tree T”. c represents each body part (each label). Then, the probability distribution is averaged with the results of all the trees, and the portion with the highest probability is the estimation result.

FIG. 7 is an explanatory diagram showing an outline of machine learning (learning processing). At the time of learning, a large amount of set images (labeled data) with region labels corresponding to distance images of various postures are prepared, and machine learning processing is performed. In the machine learning process, the offset Δ _{k of the} neighboring pixels to be compared at each node _k and the threshold value θ _k used for branch determination are determined (learned) so that the classification can be performed well by the decision tree.

  Here, as an example, a random forest is used as the machine learning method, and the difference between the distance values of the target pixel and the surrounding pixels is used as the feature amount, but other methods are also possible. For example, a method of performing multi-class classification of each pixel using a distance image as an input may be used. Further, even in the case of a random forest, feature quantities other than the difference in distance value may be used, or deep learning that performs learning including parameters corresponding to the feature quantities may be used.

  The body part identification unit 1101 may recognize the end part of the hand or foot by extracting the contour of the human body only from the distance image without using the result of machine learning. This is because, with regard to the end portions of the hands and feet, there is a high possibility that a highly accurate recognition result is obtained by contour extraction without using the result of machine learning.

  When two or more subjects S are recognized in one distance image, the body part identification unit 1101 identifies a body part for each subject S.

  The non-joint position calculation unit 1102 calculates the centroid position of each non-joint part and each end part identified by the body part identification unit 1101. The non-joint position calculation unit 1102 has the center-of-gravity positions of the non-joint parts a7, a8, a11, a12, a19, a20, a23, a24 and the end parts a15, a16, a28, a29, as indicated by x marks in FIG. Is calculated. When the body part identification unit 1101 recognizes two or more subjects S in one distance image, the non-joint position calculation unit 1102 calculates the center-of-gravity positions of each non-joint part and each end part for each subject S. calculate.

  The joint position calculation unit 1103 calculates the gravity center position of each joint part identified by the body part identification unit 1101. When the body part identification unit 1101 recognizes two or more subjects S in one distance image, the joint position calculation unit 1103 calculates the gravity center position of each joint part for each subject S. The joint position calculation unit 1103 calculates each barycentric position of the joint part and other skeleton reference positions P1, P2, and P3 as indicated by a cross mark in FIG. In this case, the joint position calculation unit 1103 may further recognize the skeleton based on the center-of-gravity positions of the joint parts and the skeleton reference positions P1, P2, and P3 as schematically illustrated in FIG.

  FIG. 11 is an explanatory diagram of an example of a method for calculating the center of gravity that can be used by the non-joint position calculation unit 1102 and the joint position calculation unit 1103. FIG. 11 shows the identification result of each part by the body part identification unit 1101, and also shows an enlarged view of X1. Here, an example in which the center of gravity position of the left shoulder (joint part) is obtained will be described with reference to an enlarged view of X1 in FIG. In the enlarged view of the part X1, each point Pi indicated by an outline “x” represents a three-dimensional coordinate corresponding to a pixel classified as a shoulder. In this case, the center-of-gravity position Joint (x, y, z) indicated by black “×” is calculated as the average of all the coordinates (x, y, z) of all Pis, as shown in Equation 3 below. You may be asked to do that. The x component and the y component are values of two-dimensional coordinates in the image plane, the x component is a horizontal component, and the y component is a vertical component. The z component represents a distance.

尚、数３の式において、“ｎ”は、各点Piの個数である。尚、３次元座標に代えて、２次元座標（ｘ，ｙ）で重心位置が算出されてもよい。

In the equation (3), “n” is the number of points Pi. Note that the gravity center position may be calculated using two-dimensional coordinates (x, y) instead of the three-dimensional coordinates.

  The recognition target person determination unit 112 determines a recognition target person (a specific individual). Here, the recognition target person is one person, but may be two or more persons. The recognition target person determination unit 112 may determine the recognition target person based on input from the user (information obtained via the input unit 107). For example, when the user wants his / her child to be a recognition target person, the user performs input to determine his / her child as the recognition target person. Alternatively, the recognition target person determining unit 112 may determine the recognition target person based on a predetermined algorithm. For example, when a person who performs suspicious behavior is detected based on a predetermined algorithm for detecting a person who performs suspicious behavior, the person may be determined as a person to be recognized. When the recognition target person determination unit 112 determines the recognition target person, the recognition target person determination unit 112 notifies the individual recognition unit 138 of the determined recognition target person.

  The subject recognition apparatus 100 further includes a skeleton data storage unit 121, a non-joint part position data storage unit 122, and a user information storage unit 124 (an example of a storage unit). The skeleton data storage unit 121, the non-joint part position data storage unit 122, and the user information storage unit 124 can be realized by the auxiliary storage unit 103 in FIG. 2, for example.

  The skeleton data storage unit 121 stores information on the centroid position of each joint part calculated by the joint position calculation unit 1103 and other skeleton reference positions P1, P2, and P3 (hereinafter referred to as “skeleton information”). .

  The non-joint part position data storage unit 122 stores information on the center-of-gravity position of each non-joint part and end part calculated by the non-joint position calculation unit 1102 (hereinafter referred to as “non-joint part information”).

  The user information storage unit 124 stores physique information related to the physique of the person to be recognized (see FIG. 12). The physique information includes length information that changes due to the physique, such as height, sitting height, shoulder width, and the like. The physique information may be numerical values such as height, sitting height, and shoulder width. In this case, the physique information may be registered (stored) in advance by user input. Alternatively, the physique information may be a distance image from the distance image sensor 21 and generated based on a distance image in which the recognition target person is the target person S. In this case, the physique information may include the centroid position of each joint part calculated by the joint position calculation unit 1103 and the centroid position of each non-joint part calculated by the non-joint position calculation unit 1102. That is, in this case, the physique information may include skeletal information and non-joint site information obtained based on a distance image in which the recognition target person is the target person S.

  In the example shown in FIG. 12, the physique information includes the height, sitting height, shoulder width, and parameters T1 and T2 related to the person to be recognized (persons A and B). The parameter T1 is the sitting height / height, and the parameter T2 is the shoulder width / height. Therefore, the parameters T1 and T2 can be calculated from the information on the height, the sitting height, and the shoulder width. When such parameters T1 and T2 are used, the sitting height and the shoulder width are normalized by the height, so that it is possible to cope with the variation in the proximity of the subject S from the distance image sensor 21. In the example illustrated in FIG. 12, the physique information is generated based on a distance image in which the recognition target person (persons A and B) is the target person S.

  The feature amount calculation unit 136 calculates a predetermined body feature amount based on the skeleton information stored in the skeleton data storage unit 121. When the body part identification unit 1101 recognizes two or more subjects S in one distance image, the feature amount calculator 136 calculates a body feature amount for each subject S.

  The predetermined body feature amount includes a distance between the two joint parts, a distance between the skeleton reference position and the joint part, or a distance between the two skeleton reference positions. The distance between the joint parts and the distance between the skeleton reference position and the joint parts are calculated based on the barycentric position of the joint parts (see the X mark in FIG. 9). The predetermined body feature amount is a feature amount that can be compared with the physique information stored in the user information storage unit 124. For example, the body feature amount related to the height is calculated as the distance between the skeleton reference position P3 of the head and the ankle a13 or a14. The body feature amount related to the sitting height is calculated as the distance between the skeleton reference position P3 of the head and the skeleton reference position P2 of the crotch (or hip joints a5 and a6). The body feature amount related to the shoulder width is calculated as the distance between the left and right shoulder joints a17 and a18. The distances related to these body feature amounts may be distances between the two-dimensional coordinates (x, y) of the respective centroid positions, or between the three-dimensional coordinates (x, y, z) of the centroid positions. It may be a distance. Alternatively, the body feature amount related to height or sitting height may be the difference (distance) of the y component of each center of gravity position, and the body feature amount related to the shoulder width is the difference (distance) of the x component of each center of gravity position. There may be.

  The predetermined body feature amount may further include a distance between the joint part and the non-joint part, or a distance between the skeleton reference position and the non-joint part. In this case, the feature amount calculation unit 136 calculates a predetermined body feature amount based further on the non-joint part information stored in the non-joint part position data storage unit 122. The distance between the joint part and the non-joint part, or the distance between the skeleton reference position and the non-joint part is calculated based on the barycentric position of the non-joint part. For example, the body feature amount related to the height is calculated as a distance between the skeleton reference position P3 of the head and the foot end part a15 or a16. The body feature amount related to the length of the hand is calculated as the distance between the shoulder joint a17 or a18 and the end portion a28 or a29 of the hand. Similarly, the distances related to these body feature amounts may be distances between the two-dimensional coordinates (x, y) of the respective gravity center positions, or the three-dimensional coordinates (x, y, z) of the respective gravity center positions. It may be a distance between. Alternatively, for example, the body feature amount related to the height may be a difference (distance) between the y components of the respective gravity center positions.

  Based on the comparison result between the physique information stored in the user information storage unit 124 and the physical feature amount calculated by the feature amount calculation unit 136, the personal recognition unit 138 may determine that the subject person S is a recognition target person. Determine. In addition, when the body part identification unit 1101 recognizes two or more target persons S in one distance image, the personal recognition unit 138 may determine whether the target person S is a recognition target person for each target person S. Determine.

  For example, based on the body feature amount α related to the height and the body feature amount β related to the sitting height calculated by the feature amount calculation unit 136, the personal recognition unit 138 has a ratio of the body feature amount β to the body feature amount α (= β / α) is calculated. Then, when the ratio β / α and the parameter T1 related to a certain recognition target person K stored in the user information storage unit 124 substantially match, the personal recognition unit 138 determines that the target person S is the recognition target person. It is determined that there is a high possibility of K. “Substantially coincidence” means that a certain amount of error is allowed, and the allowable error may be set in advance. Similarly, based on the body feature amount α related to the height calculated by the feature amount calculation unit 136 and the body feature amount γ related to the shoulder width, the individual recognition unit 138 has a ratio of the body feature amount γ to the body feature amount α ( = Γ / α) is calculated. When the ratio γ / α and the parameter T2 related to the recognition target person K stored in the user information storage unit 124 substantially match, the personal recognition unit 138 may recognize the target person S as the recognition target person K. Judgment is high. In the configuration in which both the comparison between the ratio β / α and the parameter T1 and the comparison between the ratio γ / α and the parameter T2 are performed, the individual recognition unit 138 indicates that the comparison result of both indicates that the target person S is the recognition target person K. May indicate that the subject person S is the recognition subject person K.

  FIG. 13 is a diagram illustrating a skeleton recognition result obtained from a distance image in which the person to be recognized (persons A and B) according to the physique information in FIG. The physique information in FIG. 12 is obtained from the skeleton recognition result (skeleton information) shown in FIG. As can be seen from FIGS. 12 and 13, the difference in the physique between the persons A and B appears as a difference in the body feature amount. Therefore, it can be understood that the possibility that the subject S is the recognition target can be accurately determined by using the body feature amount obtained based on the skeletal information and the non-joint part information. In FIG. 13, for reference, two-dimensional coordinates of a specific part of the person A and body feature amounts α, β, γ related to the left person are shown. As described above, the body feature amount may be calculated based on a distance between two-dimensional coordinates, a difference between x component and y component, or the like.

  According to the first embodiment, as described above, by using the body feature amount obtained based on at least one of skeletal information and non-joint site information (hereinafter referred to as “skeletal information etc.”), The possibility that S is a person to be recognized can be accurately determined. Therefore, even when two or more subjects S are recognized in one distance image, the recognition subject can be accurately identified from the two or more subjects S. Further, since the skeleton information and the like can be obtained from the distance image of the target person S with a relatively small processing load as compared with, for example, template matching by image processing, the recognition target person can be specified with high accuracy and high speed.

  Further, according to the first embodiment, as described above, when the ratio β / α and the ratio γ / α are used, the seat height and the shoulder width are normalized by the height, and thus the distance from the distance image sensor 21 is increased. It is possible to deal with fluctuations in the proximity of the target person S. That is, regardless of the distance between the distance image sensor 21 and the target person S, it is possible to accurately determine the possibility that the target person S is a recognition target person. Further, since the ratio β / α and the ratio γ / α are both based on the body feature amount α related to the longest height among the body feature amounts, the influence of errors in the body feature amounts α, β, γ is minimized. It becomes possible.

  In this way, according to the first embodiment, the possibility that the subject person S is a recognition target person can be accurately determined. Therefore, the subject person recognition system 1 can be used for personal authentication and for an unspecified number of subject persons S. The present invention can be used for various purposes such as a purpose of identifying one person to be recognized from a distance image included.

  Next, an operation example of the subject recognition apparatus 100 will be described with reference to FIG.

  FIG. 14 is a flowchart illustrating an example of processing executed by the target person recognition apparatus 100. The process illustrated in FIG. 14 may be executed offline, for example. Here, it is assumed that one distance image file in the auxiliary storage unit 103 or the recording medium 105 is a processing target, and the number of distance image frames included in the processing target distance image file is “N”.

  In step S1400, the recognition target person determining unit 112 determines a recognition target person. At this time, the recognition subject person determination unit 112 may check whether or not the physique information related to the recognition subject person is stored in the user information storage unit 124. When the physique information related to the recognition target person is not stored in the user information storage unit 124, the recognition target person determination unit 112 outputs an error message or the like to the display device (not shown), You may prompt the input of the physique information concerning a recognition subject person. As described above, the input of the physique information related to the recognition target person may be realized by inputting a distance image in which the recognition target person is captured by only one person.

  In step S1402, body part position calculation unit 110 sets “j” to “1”.

  In step S <b> 1404, the body part position calculation unit 110 acquires a distance image of the “j” th frame from the auxiliary storage unit 103 or the recording medium 105.

  In step S1406, the body part position calculation unit 110 identifies each part of the body of the subject S in the distance image based on the distance image acquired in step S1404, and generates skeleton information and the like. The method for generating the skeleton information and the like is as described above. When the body part position calculation unit 110 recognizes two or more subjects S in the distance image acquired in step S1404, the body part position calculation unit 110 sets a flag F indicating that to “1”. The initial value of the flag F is “0”.

  In step S1408, the feature amount calculation unit 136 determines whether the body part position calculation unit 110 has recognized two or more subjects S in the distance image. That is, the feature amount calculation unit 136 determines whether or not the flag F is “1”. If the determination result is “YES”, the process proceeds to step S1410, and otherwise, the process proceeds to step S1416.

  In step S1410, the feature amount calculation unit 136 calculates a predetermined body feature amount for each subject S. In FIG. 14, the feature amount calculation unit 136 calculates, for each subject S, a body feature amount α related to height, a body feature amount β related to sitting height, and a body feature amount γ related to shoulder width, and based on the calculation result. The ratios of the body feature amounts β and γ to the body feature amount α (= β / α, γ / α) are calculated.

  In step S1412, the individual recognizing unit 138 for each subject S, the physique information related to the recognition subject determined in step S1400, and the ratios (= β / α, γ / α) obtained in step S1410. Based on the above, the possibility that the subject person S is a recognition subject person is determined. Specifically, the personal recognition unit 138 substantially matches the ratio β / α and the parameter T1 related to the recognition target person, and substantially matches the ratio γ / α and the parameter T2 related to the recognition target person. In this case, it is determined that the target person S is a recognition target person. The parameters T1 and T2 are included in the physique information as described above, and are seat height / body feature amount and shoulder width / height, respectively. If the determination result is “YES”, the process proceeds to step S1414; otherwise, the process proceeds to step S1416.

  In step S1414, the personal recognition unit 138 associates the skeleton information related to the target person S determined to be the recognition target person among the skeleton information related to the plurality of target persons S with the recognition target person. For example, the personal recognition unit 138 assigns an identifier related to the recognition target person to the skeleton information related to the target person S determined to be the recognition target person.

  In step S1416, the personal recognition unit 138 determines whether “j” is “N (number of frames of the distance image to be processed)”. If j = N, the process ends as it is. If j ≠ N, the process returns to step S1404 via step S1418.

  In step S1418, the personal recognition unit 138 increments “j” by “1”.

  According to the process shown in FIG. 14, the person to be recognized can be identified in the N-frame distance image stored in the auxiliary storage unit 103 or the recording medium 105. Thereby, for example, the user can easily extract only the distance image including the recognition target person from the N frame distance images. In addition, when N frames of distance images are continuous in time (that is, when they are moving images), it is easy to extract only the movement of the person to be recognized.

  The process shown in FIG. 14 is executed offline, but may be executed, for example, online (in real time) and at predetermined intervals (for example, every frame period). In this case, step S1402, step S1416, and step S1418 are omitted, and the body part position calculation unit 110 may acquire a distance image directly from the distance image sensor 21 in step S1404.

  In the process shown in FIG. 14, if the determination result in step S1408 is “NO”, the process proceeds to step S1416, but is not limited thereto. When the determination result of step S1408 is “NO”, the feature amount calculation unit 136 may calculate each ratio (= β / α, γ / α) related to one target person S. In this case, the personal recognition unit 138 determines the possibility that one target person S is a recognition target person based on the physique information related to the recognition target person and each ratio (= β / α, γ / α). To do. When the determination result is “YES”, the personal recognition unit 138 associates the skeleton information related to the target person S determined to be the recognition target person with the recognition target person.

[Example 2]
The subject recognition system 1A according to the second embodiment is different in that robot control data is generated using information obtained by subject recognition processing. In the following, components that may be the same as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted.

  The subject recognition system 1A includes a distance image sensor 21 and a subject recognition device 100A.

  The subject recognition device 100A may have the hardware configuration shown in FIG. FIG. 15 is a block diagram illustrating an example of functions of the subject recognition device 100A. FIG. 15 also illustrates a distance image sensor 21 that inputs a distance image to the subject recognition device 100A.

  FIG. 15 is a block diagram illustrating an example of functions of the subject recognition device 100A. FIG. 15 also illustrates a distance image sensor 21 that inputs a distance image to the subject recognition device 100A.

  The subject recognition apparatus 100A is different from the subject recognition apparatus 100 according to the first embodiment described above in that it further includes a robot control data generation unit 140. The robot control data generation unit 140 can be realized by the control unit 101 illustrated in FIG. 2 executing one or more programs stored in the main storage unit 102.

  The robot control data generation unit 140 generates robot control data based on the skeleton information generated by the body part position calculation unit 110 and the recognition result by the personal recognition unit 138. The robot control data generation unit 140 generates robot control data so that the robot moves in accordance with the movement of the person to be recognized.

  FIG. 16 is a diagram illustrating an example of the robot 90. In FIG. 16, the skeleton 910 inside the robot 90 is schematically shown in perspective. The robot 90 is, for example, a humanoid, and has, for example, a skeleton 910 similar to the skeleton of a human body (see FIG. 10). Therefore, the robot 90 shown in FIG. 16 can imitate the movement of a person.

  Next, an operation example of the subject recognition apparatus 100A will be described with reference to FIG.

  FIG. 17 is a flowchart illustrating an example of processing executed by the target person recognition apparatus 100A. The process illustrated in FIG. 17 may be executed offline, for example.

  The process shown in FIG. 17 is different from the process shown in FIG. 14 in the first embodiment in that step S1700 is added. Below, a different point is demonstrated.

  If the determination result in step S1416 is “YES” (ie, j = N), the process proceeds to step S1700.

  In step S1700, the robot control data generation unit 140 obtains the robot control data based on the recognition result (the skeleton information related to the target person S determined to be a recognition target person) by the personal recognition unit 138 obtained in step S1414. Generate. For example, the robot control data generation unit 140 sets the robot control data for each frame so that each joint position of the skeleton 910 of the robot 90 corresponds to the skeleton information related to the target person S determined to be the recognition target person. Generate. In this case, the robot control data is time-series control data over a plurality of frames, and the movement of the robot 90 according to the movement of the person to be recognized can be realized.

  According to the processing shown in FIG. 17, the person to be recognized can be identified in the N-frame distance image stored in the auxiliary storage unit 103 or the recording medium 105. As a result, as described above, the robot control data corresponding to the recognition target person can be generated, and the movement of the robot 90 in accordance with the movement of the recognition target person can be realized.

  Note that the processing shown in FIG. 17 is executed offline, but may be executed, for example, online (in real time) at every predetermined cycle (for example, every frame cycle). In this case, step S1402, step S1416, and step S1418 are omitted, and the body part position calculation unit 110 may acquire a distance image directly from the distance image sensor 21 in step S1404.

  Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes can be made within the scope described in the claims. It is also possible to combine all or a plurality of the components of the above-described embodiments.

  For example, in the above-described embodiment, the subject recognition apparatus 100 includes the non-joint position calculation unit 1102 and the non-joint part position data storage unit 122, but the non-joint position calculation unit 1102 and the non-joint part position data storage unit 122 include It may be omitted. In this case, the body feature amount is always derived based on the skeleton information stored in the skeleton data storage unit 121.

  Further, in the above-described embodiments, the ratio β / α and the ratio γ / α are used, but equivalently, the ratio α / β and the ratio α / γ are used instead of the ratio β / α and the ratio γ / α. May be.

In addition, the following additional remarks are disclosed regarding the above Example.
[Appendix 1]
Based on the skeletal information obtained from the first distance image using the distance information of the subject as a pixel value, a first distance corresponding to the first body feature of the subject and another second body feature of the subject A second distance corresponding to the quantity,
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition method executed by a computer, comprising: determining.
[Appendix 2]
The object according to appendix 1, wherein the physique information is a second distance image different from the first distance image, and is generated based on a second distance image having pixel information as distance information of the specific individual. Recognition method.
[Appendix 3]
The first distance and the second distance are respectively calculated based on the coordinates of the first body part on the first distance image and the coordinates of another second body part on the first distance image. And
In the case of calculating the first distance, the first body part is a head, and the second body part is a part ahead of an ankle or an ankle. Recognition method.
[Appendix 4]
The subject recognition method according to appendix 3, wherein the first distance and the second distance are calculated based on a centroid position of the first body part and a centroid position of the second body part, respectively. .
[Appendix 5]
The second distance corresponds to the sitting height;
The subject recognition method according to appendix 3 or 4, wherein when calculating the second distance, the first body part is a head, and the second body part is a crotch part.
[Appendix 6]
The second distance corresponds to a shoulder width;
The subject recognition method according to appendix 3 or 4, wherein when calculating the second distance, the first body part is a left shoulder part, and the second body part is a right shoulder part. .
[Appendix 7]
The target person recognition method according to claim 1, wherein the target person is a plurality.
[Appendix 8]
The subject recognition method according to any one of appendices 1 to 7, further including generating data for controlling the robot based on the skeleton information and the determination result of the possibility.
[Appendix 9]
Based on the skeletal information obtained from the distance image using the distance information of the subject person as a pixel value, the first distance corresponding to the first body feature amount of the subject person and another second body feature amount of the subject person A calculation unit for calculating a corresponding second distance;
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition device including a determination unit for determining.
[Appendix 10]
A distance image sensor that generates a distance image having the distance information of the subject as pixel values;
A subject recognition device,
The subject recognition device
Based on the skeletal information obtained from the distance image, a first distance corresponding to the first body feature of the subject and a second distance corresponding to another second body feature of the subject are calculated. A calculation unit;
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition system including a determination unit for determining.
[Appendix 11]
Based on the skeletal information obtained from the distance image using the distance information of the subject person as a pixel value, the first distance corresponding to the first body feature amount of the subject person and another second body feature amount of the subject person Calculating the corresponding second distance,
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. judge,
A subject recognition program that causes a computer to execute processing.

DESCRIPTION OF SYMBOLS 1, 1A Target person recognition system 21 Distance image sensor 90 Robot 100, 100A Target person recognition apparatus 110 Body part position calculation part 112 Recognition target person determination part 121 Skeletal data storage part 122 Non-joint part position data storage part 124 User information storage part 136 Feature amount calculation unit 138 Personal recognition unit 140 Robot control data generation unit 1101 Body part identification unit 1102 Non-joint position calculation unit 1103 Joint position calculation unit

Claims (8)

Based on the skeletal information obtained from the first distance image using the distance information of the subject as a pixel value, a first distance corresponding to the first body feature of the subject and another second body feature of the subject A second distance corresponding to the quantity,
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition method executed by a computer, comprising: determining.   The physique information is a second distance image different from the first distance image, and is generated based on a second distance image having pixel information as distance information of the specific individual. Target person recognition method. The first distance and the second distance are respectively calculated based on the coordinates of the first body part on the first distance image and the coordinates of another second body part on the first distance image. And
The target according to claim 1 or 2, wherein when calculating the first distance, the first body part is a head, and the second body part is an ankle or an anterior part of the ankle. Recognition method. The second distance corresponds to the sitting height;
The subject recognition method according to claim 3, wherein, when calculating the second distance, the first body part is a head, and the second body part is a crotch part. The second distance corresponds to a shoulder width;
The subject recognition method according to claim 3, wherein, when calculating the second distance, the first body part is a left shoulder part, and the second body part is a right shoulder part. Based on the skeletal information obtained from the distance image using the distance information of the subject person as a pixel value, the first distance corresponding to the first body feature amount of the subject person and another second body feature amount of the subject person A calculation unit for calculating a corresponding second distance;
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition device including a determination unit for determining. A distance image sensor that generates a distance image having the distance information of the subject as pixel values;
A subject recognition device,
The subject recognition device
Based on the skeletal information obtained from the distance image, a first distance corresponding to the first body feature of the subject and a second distance corresponding to another second body feature of the subject are calculated. A calculation unit;
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. A subject recognition system including a determination unit for determining. Based on the skeletal information obtained from the distance image using the distance information of the subject person as a pixel value, the first distance corresponding to the first body feature amount of the subject person and another second body feature amount of the subject person Calculating the corresponding second distance,
Based on the ratio of the first distance and the second distance and the physique information acquired from the storage unit that stores the physique information related to the physique of a specific individual, the possibility that the target person is the specific individual. judge,
A subject recognition program that causes a computer to execute processing.

Images