JP6819633B2 - Personal identification device and feature collection device - Google Patents

Description

The present invention relates to a personal identification technique for identifying a person in a captured image.

There is a technology that uses face recognition as an individual identification technology, but it cannot be identified if the subject does not face the camera. In addition, there is also a problem that it is difficult to distinguish the faces in the home because they have genetically similar faces. Therefore, it is conceivable to make it possible to identify even in a scene where the face cannot be photographed by performing identification using human body characteristics (posture, silhouette) and behavioral characteristics (traffic line pattern, place of stay).

In order to perform identification using human body characteristics and behavioral characteristics, it is necessary to collect and learn human body characteristics and behavioral characteristics for each subject. In addition, behavioral characteristics may change according to changes in time and environment, so it is desirable to continuously collect these characteristics and update the registration information used for identification to respond to the changes. ..

In order to collect human body characteristics and behavioral characteristics, it is necessary to identify who the detected person is, but there are cases where accurate personal identification cannot be performed only by identification based on images. As described above, face recognition cannot be used unless the face can be photographed, and if the learning of human body features and behavioral features is insufficient, the identification based on these features is not accurate.

Patent Document 1 discloses that a call voice is output to a person and the optimum face to be registered is selected based on the change of the face before and after the call voice output. For example, it is disclosed to select a face that turns around in response to the call. However, Patent Document 1 only discloses a method of acquiring a face image for learning, and cannot deal with the problem that accurate personal identification cannot be performed unless an image showing a face is obtained at the time of identification. ..

Japanese Unexamined Patent Publication No. 2013-182325

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a personal identification technique capable of accurately identifying a person in an image.

In order to achieve the above object, the present invention employs a method of identifying a person based on identification based on an image and identification based on a response voice to an output voice.

Specifically, the first aspect of the present invention includes a voice output means, a voice input means, an image input means, a detection means, and a person identification means. The audio output means outputs audio. The voice input means acquires a voice that responds to the output voice. The image input means acquires a moving image. The detection means detects the human body from the input moving image. Any method can be adopted as the human body detection method. The person identification means includes a first identification means for performing identification based on an image and a second identification means for performing identification based on voice. As the first identification means, for example, an identification method using facial features, human body features (posture or silhouette), and behavioral features (traffic line pattern or place of stay) can be adopted, but if the identification is based on the features obtained from the image. Other identification methods may be adopted. As the second identification means, the identification means based on the acoustic feature amount obtained by the waveform analysis of the input voice or the feature amount of the word or sentence obtained by the natural language analysis can be adopted, but other identification means can be adopted as long as the identification is based on the input voice. The method may be adopted. The person identification means identifies a detected person based on the identification result by the first identification means (hereinafter, the first identification result) and the identification result by the second identification means (hereinafter, the second identification result).

According to such a configuration, even if it is not possible to accurately identify a person based only on an image, it is possible to accurately identify a person by making a judgment together with an identification result based on voice.

In this embodiment, the person identification means may be configured to perform identification in consideration of the second identification means when the first identification means cannot identify with high reliability. That is, when the reliability by the first identification means is less than the first threshold value, the person identification means performs identification by the second identification means and is based on the first identification result and the second identification result. It may be configured to identify the person. In the identification based on the first identification result and the second identification result, the person identification means determines the identification result when the first identification result and the second identification result match, and when the two identification results are different. May leave the identification result undetermined.

Further, in the present embodiment, when the reliability by the first identification means is equal to or higher than the first threshold value, the first identification result may be used as the identification result of the person.

According to such a configuration, when the identification can be performed with high reliability only from the image, the processing amount can be reduced, and it is not necessary to oblige the user to respond by omitting the question by voice.

In this embodiment, the audio output from the audio output means may be performed at a predetermined timing. The predetermined timing is the timing when the identification reliability of the person becomes less than the threshold value, the timing when the first predetermined time elapses from the timing when the person is detected, and the second predetermined time when there is almost no time change of the person. It can be at least one of a continuous timing, a timing when the person goes out of the imaging range, and a timing when the distance between the information processing device and the person becomes equal to or less than a predetermined distance. The identification reliability of a person may be the reliability of identification by the first identification means, or may be the reliability of integrating the identification results of the first identification means and the second identification means. The first predetermined time may be a fixed time, or may be a time determined according to the number of acquisitions of the feature and the amount of data of the feature when the feature is acquired from the image.

In this embodiment, it is preferable that the person identification means determines the content of the output voice according to the reliability of the first identification means. For example, when the reliability is less than the second threshold value, the person identification means outputs the content including the name of the person in the first identification result or the content inquiring who the person is. Can be determined as the content of. The reliability may be divided into three or more levels, and the content of the output voice may be determined according to each level. For example, when dividing the reliability into three levels, if the level is high, the content does not include the name of the person, if the level is medium, the content includes the name of the person, and if the level is low, the content is directly. It may be the content to inquire who the person is. As an example, at a high level, "What are you doing there?", At a medium level, "Mom, what are you doing there?", And at a low level, "Who is there?" Can be adopted.

Further, in the present embodiment, when the first identification result and the second identification result do not match, the person identification means newly outputs a voice and is based on an input voice that responds to the new output voice. Identification may be performed by the second identification means. In this case, the content of the new output voice may be the content of confirming the person more directly than the content of the previous output voice. If the first identification result and the second identification result do not match, it is not possible to determine who the person is, so voice output is performed to more directly confirm who the person is. Then, it is good to identify who the person is from the response.

A second aspect of the present invention includes the personal identification device, a feature acquisition means for acquiring at least one of the detected features related to the human body or behavior of a person from a moving image input to the image input means. It is a feature collecting device provided with a feature registering means for registering a feature acquired by the feature acquiring means in association with a person identified by the person identifying means. The features collected in this way can be used for learning the classifier.

The present invention can be regarded as an information processing system having at least a part of the above-mentioned configuration or function. Further, the present invention provides a method for controlling an information processing method or an information processing system including at least a part of the above processing, a program for causing a computer to execute these methods, or a program such as such. It can also be regarded as a recorded computer-readable recording medium. Each of the above configurations and processes can be combined with each other to construct the present invention as long as there is no technical contradiction.

According to the present invention, a person in an image can be identified with high accuracy.

FIG. 1 is a block diagram showing a configuration example of a personal identification device to which the present invention is applied. FIG. 2 is a block diagram showing a configuration example of the feature collecting device according to the first embodiment. FIG. 3 is a flowchart showing an example of the feature collection process according to the first embodiment. FIG. 4 is a flowchart showing an example of the utterance content determination process in the first embodiment. 5A and 5B are diagrams for explaining the skeleton information of the human body detected from the input image and the posture detection based on the skeleton. FIG. 6 is a flowchart showing an example of the utterance content determination process in the second embodiment.

<Application example>
An application example of the present invention will be described. When collecting personal and behavioral characteristics of an individual, it is necessary to identify who the person being collected is. Face recognition is an image-based personal identification method, but it is not always possible to photograph a face, so accurate identification may not be possible. In addition, although identification based on human body characteristics and behavioral characteristics is possible, it is not realistic to rely entirely on identification based on human body characteristics and behavioral characteristics in collecting characteristics for identification based on human body characteristics and behavioral characteristics.

FIG. 1 is a block diagram showing a configuration example of the personal identification device 10 to which the present invention is applied. The personal identification device 110 identifies a person in the moving image from the input moving image or sound. The personal identification device 10 includes an image input unit 11, a human body detection unit 12, a person identification unit 13, a voice output unit 14, and a voice input unit 15.

The image input unit 11 acquires the captured moving image (moving image data). For example, the image input unit 11 is an input terminal into which image data is input. The image input unit 11 is an example of the image acquisition means of the present invention.

The human body detection unit 12 detects a region (human body region) that is typical of the human body from the frame image to be processed of the moving image data. Human body detection may be performed using a detector learned by any existing algorithm. The human body detection unit 12 is an example of the detection means of the present invention.

The voice output unit 14 outputs the utterance content obtained from the person identification unit 13 as voice data. The audio output unit 14 is an example of the audio output means of the present invention.

The voice input unit 15 acquires voice data. The voice input unit 15 is an example of the voice input means of the present invention.

The person identification unit 13 identifies a person in the moving image based on the moving image acquired by the image input unit 11 and the sound acquired by the voice input unit 15. The person identification unit 13 also determines the content of the utterance for drawing out the response voice used for voice identification and the timing of the utterance. The person identification unit 13 is an example of the person identification means of the present invention.

The first identification unit 131 identifies a person detected by the human body detection unit 12 from the moving image acquired by the image input unit 11. The first identification unit 131 identifies a person based on the input image. Specifically, the detected person is identified based on at least one of facial features, human body features, and behavioral features in the image. The first identification unit 131 is an example of the first identification means of the present invention.

The second identification unit 133 identifies the person detected by the human body detection unit 12 based on the voice acquired by the voice input unit 15. The second identification unit 133 may perform identification based on the acoustic feature amount obtained by performing the waveform analysis of the voice, or may perform the identification based on the language feature amount obtained by performing the natural language analysis. .. Identification based on the acoustic feature amount can be performed based on the degree of agreement with the acoustic feature amount registered in advance. Identification based on language features can be performed based on the meaning of the content of the output voice and the content of the response voice. The second identification unit 133 is an example of the second identification means of the present invention.

The person identification unit 13 determines the timing of utterance from the voice output unit 14 and the content of the utterance sentence. In this application example, the utterance (voice output) is performed from the voice output unit 14 in order to obtain a voice response from the detecting person and perform voice-based detection. Therefore, the person identification unit 13 determines the timing at which it is assumed that the identification based on the voice needs to be performed as the utterance timing. The person identification unit 13 may typically determine a timing at which the reliability of identification by the first identification unit 131 is low (less than the threshold value TH1) as the utterance timing. In addition, the person identification unit 13 periodically speaks, speaks when there is no movement of the person, and speaks when the identification results of the first identification unit 131 and the second identification unit 133 are different. You may decide the utterance timing so as to do so. The content of the utterance may be determined according to the reliability of the person identification. For example, if the reliability is high, the utterance of natural communication is performed, and if the reliability is low, the utterance including the name of the estimated person or the utterance that directly inquires who the detected person is. It is conceivable to do. Here, the utterance including the name of the person and the utterance inquiring who the person is are examples of the utterances for confirming who the person is.

It should be noted that the utterance and the response may be performed a plurality of times, and the second identification unit 133 may identify the response voices a plurality of times. In this case, for example, the first utterance of natural communication is performed, and the second identification unit 133 performs identification based on the acoustic characteristics of the response voice. Here, when the identification results of the first identification unit 131 and the second identification unit 133 match, or when the reliability of the identification result of the second identification unit 133 is high, it is considered that the person is correctly identified. Therefore, it is not necessary to make a second utterance, and it is possible to continue natural communication. On the other hand, if the identification result of the second identification unit 133 based on the first response is different from the identification result of the first identification unit 131, or the identification reliability is low, the content of the second utterance is the first utterance. It is better to confirm the person more directly in comparison with. In the second time, the person can be identified by the content of the response (semantic interpretation by natural language analysis), so it is expected that a highly reliable identification result can be obtained.

As described above, according to the personal identification device 10 according to the present application example, by identifying the person in the image based on the two identification methods of the image and the sound, the identification cannot be performed with high reliability only from the image. However, it can be identified accurately. In addition, since it is possible to minimize the number of voice questions for personal identification, it is possible to suppress the user from feeling annoyed. Further, by determining the utterance content as described above, natural communication is possible even when asking a question to the user, and from such a point as well, it is possible to suppress the user from feeling annoyed.

(First Embodiment)
The first embodiment of the present invention is a feature collecting device that collects human body features and behavioral features obtained by photographing a person, and is mounted on a home-use communication robot 1 (hereinafter, also simply referred to as a robot 1). .. The feature collector collects these features for learning personal identification.

[Constitution]
FIG. 2 is a diagram showing the configuration of the robot 1. The robot 1 includes a feature collecting device 100, a camera 200, a speaker 300, and a microphone 400. The robot 1 has a processor (arithmetic device) such as a CPU, a main storage device, an auxiliary storage device, a communication device, and the like, and each process of the feature collection device 100 is executed when the processor executes a program. Will be done.

The camera 200 continuously shoots with visible light or non-visible light (for example, infrared light), and inputs the captured image data to the feature collecting device 100. The imaging range of the camera 200 may be fixed or variable. The imaging range of the camera 200 may be changed by changing the orientation of the camera 200, or by autonomously moving the robot 1.

The speaker 300 converts the voice data input from the feature collecting device 100 into an acoustic wave and outputs it. The microphone 400 converts an acoustic wave such as a voice spoken by the user into voice data and outputs the sound wave to the feature collecting device 100. The microphone 400 may be configured as a microphone array so that the sound sources can be separated.

From the image captured by the camera 200, the feature collecting device 100 acquires at least one of the posture, gesture, silhouette, flow line, and staying place of the person in the image as the feature of the person. Further, the feature collecting device 100 identifies a person in the image and stores the feature information in association with the individual. In order to perform such an operation, the feature collecting device 100 includes a personal identification device 110, a feature acquisition unit 120, and a feature registration unit 130.

The personal identification device 110 identifies a person in the moving image from the moving image input from the camera 200 and the sound input from the microphone 400. The personal identification device 110 includes an image input unit 111, a human body detection unit 112, a person identification unit 113, a voice output unit 114, and a voice input unit 115. The person identification unit 113 includes a first identification unit 1131, an utterance control unit 1132, a second identification unit 1133, and a person identification unit 1134.

The image input unit 111 is an interface for receiving image data from the camera 200. In the present embodiment, the image input unit 111 receives the image data directly from the camera 200, but the image data may be received via the communication device or the image data via the recording medium. ..

The human body detection unit 112 detects the human body region from the input image. Any existing algorithm for detecting the human body can be adopted. Further, the human body detection unit 112 may detect the human body once detected by a tracking process. At the time of tracking, the human body detection unit 112 stores the type, shape, color, etc. of the clothes and ornaments of the person once detected, and these are stored for a predetermined time (for example, about 1 hour to several hours). It may be detected using features. In addition, the type, shape, and color of clothes and accessories may be used for identification by the first identification unit 1131. These features can be useful for tracking, as clothing is unlikely to change in a short period of time.

The person identification unit 113 identifies (identifies) who the person (detection person) detected by the human body detection unit 112 is.

The first identification unit 1131 identifies the detected person based on the image acquired by the image input unit 11. Specifically, the detected person is identified based on at least one of facial features, human body features, and behavioral features in the image. In order to perform these identifications, the facial features, human body features, and behavioral features of each person to be identified are registered in advance. As described above, when the human body detection unit 112 has acquired the type, shape, and color of the clothes and ornaments of the detected person, these characteristics may also be used for identification. In addition, the first identification unit 1131 also calculates the reliability of the identification result (an index indicating how probable the identification result is).

The utterance control unit 1132 determines the timing of utterance from the voice output unit 114 and the content of the utterance sentence. The reason why the utterance (voice output) is performed from the voice output unit 114 is to obtain a voice response from the detecting person and perform detection based on the voice. Therefore, the person identification unit 113 determines the timing at which it is assumed that the identification based on the voice needs to be performed as the utterance timing. In addition, the utterance content should be content that the user does not feel annoyed and that can be identified based on the response voice. The specific method of determining the utterance timing and the utterance content will be described later.

The second identification unit 1133 identifies the person detected by the human body detection unit 12 based on the voice acquired by the voice input unit 15. The second identification unit 1133 may perform the identification based on the acoustic features obtained by performing the waveform analysis of the voice, or may perform the identification based on the language features obtained by performing the natural language analysis. .. Identification based on the acoustic feature amount can be performed based on the degree of agreement with the acoustic feature amount registered in advance. Identification based on language features can be performed based on the meaning of the content of the output voice and the content of the response voice. The second identification unit 1133 also calculates the reliability of the identification result.

The person identification unit 1134 finally determines who the detected person is based on the identification result (first identification result) of the first identification unit 1131 and the identification result (second identification result) of the second identification unit 1133. Identify. For example, when the first identification result and the second identification result match, the person identification unit 1134 may use the identification result as the final result, or adopts either the first identification result or the second identification result. You may. The person identification unit 1134 may specify who the detection person is in consideration of the identification reliability of the first identification unit 1131 and the identification reliability of the second identification unit 1133. For example, when the identification reliability of the first identification unit 1131 is sufficiently high, the identification result of the first identification unit 1131 may be the final result without performing the identification by the second identification unit 1133. The person identification unit 1134 also calculates the reliability of the identification result. This reliability is used, for example, in determining the utterance timing and the utterance content by the utterance control unit 1132.

The voice output unit 114 acquires text data of the utterance voice from the utterance control unit 1132, converts it into voice data by voice synthesis processing, and outputs it from the speaker 300.

The voice input unit 115 receives a voice signal from the microphone 400, converts it into voice data, and outputs it to the second identification unit 1133. The voice input unit 115 may perform preprocessing such as noise removal and speaker separation.

The feature acquisition unit 120 acquires the characteristics of the person detected by the human body detection unit 112 from the input image. The feature acquired by the feature acquisition unit 120 is, for example, at least one of a feature related to the human body (for example, a feature related to a part, skeleton, posture, and silhouette of the human body) and a feature related to behavior (features related to gestures, flow lines, and staying place). including. Hereinafter, the features acquired by the feature acquisition unit 120 are also referred to as human body / behavioral features. When these features have already been calculated by the human body detection unit 112 or the first identification unit 1131, the feature acquisition unit 120 may acquire the calculated features. On the contrary, the feature calculated by the feature acquisition unit 120 may be used by the first identification unit 1131.

The acquisition of human body / behavioral characteristics will be described with reference to FIGS. 5A and 5B. The feature acquisition unit 120 acquires skeleton information indicating the skeleton of the detected person 51 from the input image 50. The skeleton information is acquired using, for example, OpenPose. The skeletal information is also characteristic information indicating the human body, and is also characteristic information indicating parts of the human body (head, neck, shoulders, elbows, hands, hips, knees, ankles, eyes, ears, fingertips, etc.). In FIG. 5, the skeleton (skeleton information) 52 of the detection person 51 is detected from the image 50.

As shown in FIG. 5B, the shape of the skeleton depends on the posture. Therefore, the feature acquisition unit 120 can acquire the posture information of the person from the positional relationship (skeleton information) of the parts of the human body. The posture information may be information listing the relative positional relationships of each part of the human body, or may be the result of classifying the positional relationships of the parts of the human body. Examples of posture classifications include upright, stoop, O-legs, and X-legs. Further, the feature acquisition unit 120 may acquire the silhouette information of the detected person, or may obtain the posture from the silhouette information.

The above-mentioned skeletal information, posture information, and silhouette information all correspond to the human body features in the present invention.

In addition, the feature acquisition unit 120 can detect motion information (gesture) from changes in posture information of each frame. As the detection result of the gesture, for example, information indicating walking, bending and stretching, lying down, arms folded, and the like can be obtained. The positional relationship between the upper arm and the forearm differs between the case where the arms are folded and the case where the arms are not folded. In this way, the positional relationship of each part depends on the gesture. Therefore, the gesture can be detected from the positional relationship of each part based on the skeleton information. Gestures accompanied by movements such as walking and bending and stretching may be detected by using a plurality of skeletal information corresponding to a plurality of images captured at different times. For walking, information indicating the ratio of stride length to shoulder width may be obtained.

Further, the feature acquisition unit 120 may acquire the staying place and the flow line (movement route) of the detected person as features. The characteristics related to the place of stay can be obtained by calculating the ratio of the staying time to the length of the predetermined period (stay rate) for each stay determination area based on the time change of the person's position in the predetermined period such as the past several minutes. .. The detection result of the staying place is obtained in the form of a staying map (heat map) showing the staying rate of each staying determination area. The feature related to the flow line is the information showing the stay position in chronological order, and can be obtained by the same method as the stay map.

The above-mentioned information on movements, places of stay, and flow lines all correspond to behavioral features in the present invention.

The feature registration unit 130 associates the human body / behavioral features acquired by the feature acquisition unit 120 with the person identified by the personal identification device 110 (person identification unit 113) and registers them in a storage unit such as a collation database. The timing at which the feature registration unit 130 registers may be arbitrary, but may be, for example, a timing at which the person identification unit 113 can perform identification with high reliability or a timing at which the tracking of the person is completed.

[processing]
FIG. 3 is a flowchart showing the overall flow of the feature collection process performed by the feature collection device 100. Hereinafter, the feature collection process in the present embodiment will be described with reference to FIG. It should be noted that this flowchart conceptually explains the feature collection process in the present embodiment, and it is not necessary to implement the process according to this flowchart in the embodiment.

In step S10, the human body detection unit 112 detects the human body from the input image. If the human body is not detected (S11-NO), the process returns to step S10 to detect the human body from the next frame image to be processed. On the other hand, when the human body is detected (S11-YES), the process proceeds to step S12.

In step S12, the feature acquisition unit 120 acquires the human body / behavioral features of the detected person.

In step S13, the first identification unit 1131 performs personal identification based on the human body / behavioral characteristics. The face feature may also be acquired in step S12, and the first identification unit 1131 may perform personal identification based on the face feature in step S13.

Here, the reliability of the identification based on the image will be briefly described. It can be expected that identification based on facial features can be performed with relatively high accuracy (high reliability), but it can be performed only when the user is facing the robot 1. On the other hand, identification based on human body characteristics and behavioral characteristics can be performed if the user's body is shown, but the accuracy is not always good. In particular, it is assumed that highly reliable identification cannot be performed at the stage of collecting features for initial learning of human body / behavioral features.

The first identification unit 1131 continuously identifies the person included in the moving image, and calculates the current identification reliability by totaling the identification reliability in each frame. At this time, the first identification unit 1131 may use a weighted average that gives a large weight to the reliability of the recent identification as the current identification reliability.

In step S14, the utterance control unit 1132 determines whether or not it is time to speak. The timing of utterance may be set as a condition in advance, and in step S14, the person identification unit 13 may determine whether or not the set condition is met.

The following can be adopted as conditions for utterance.
(1) Time For example, the first utterance is made after a predetermined time (for example, 10 minutes) has elapsed from the detection of the human body, and then the utterance is made at a predetermined interval.
(2) Data amount For example, when a feature of a predetermined data amount (for example, data for 100 times) is acquired, an utterance is performed.
(3) Behavior stop When the behavior of the detected person does not change for a certain period of time. For example, it corresponds to the case where a certain period of time has passed after sitting on the sofa and starting watching TV.
(4) Movement outside the imaging range When it is predicted that the detected person will move outside the imaging range. For example, it corresponds to the case where the detected person moves or goes out from the current room to another room.
(5) Situation where it is easy to speak When the robot reaches a situation suitable for interacting with the detected person. For example, when the detected person and the robot are facing each other (the robot can detect the face of the detected person), and the distance between the detected person and the robot is within a predetermined distance (for example, 3 meters).
(6) When the identification reliability is low The person identification unit 113 determines the final identification result by using the identification results of both the first identification unit 1131 and the second identification unit 1133. Therefore, if the identification reliability by the first identification unit 1131 is equal to or higher than the threshold value TH1, the result is determined as the identification result of the person identification unit 113, and if the identification reliability is less than the threshold value TH1, the second identification unit 1133 determines the result. The utterance may be made to identify. Alternatively, when the identification reliability is less than the threshold value TH1 after considering the identification results of both the first identification unit 1131 and the second identification unit 1133, an utterance is made to further identify by the second identification unit 1133. You may do so.

A plurality of the above conditions may be combined, and for example, the utterance may be made when any one of the plurality of conditions is satisfied, or the utterance may be made when some of the plurality of conditions are satisfied at the same time. You may try to do it. Furthermore, in situations where it is not appropriate to speak, such as when the detected person is sleeping or concentrating, the above conditions may be met but the person may not speak.

Since the purpose of the utterance here is to obtain a response voice for personal identification, the utterance is made at the above timing, but the utterance at other timings is prohibited. It's not a thing. For example, the user may be talked to for communication at a timing when the above conditions are not satisfied.

If it is determined in step S14 that the utterance timing condition is satisfied, the process proceeds to step S15, and if not, the process proceeds to step S19.

In step S15, the utterance control unit 1132 determines the content of the utterance (speech text). In the present embodiment, the utterance control unit 1132 determines the utterance content based on the current identification reliability. FIG. 4 is a flowchart illustrating the utterance content determination process in step S15.

As shown in FIG. 4, in step S131, the utterance control unit 1132 determines the utterance level according to the current identification reliability. In the present embodiment, for example, a high reliability with an identification reliability of 0.8 or more, a medium reliability with an identification reliability of 0.5 or more and less than 0.8, and a low reliability with an identification reliability of less than 0.5 Classify into 3 levels. This threshold value is merely an example and may be appropriately determined according to the system requirements. Further, the threshold value may change depending on the situation. Moreover, although the level is divided into three stages in this embodiment, it may be divided into two stages or four or more stages.

If the identification reliability is high, the process proceeds to step S152, and the utterance control unit 1132 determines the utterance content with an emphasis on the naturalness of communication. For example, the utterance control unit 1132 determines the utterance content that does not include the name of the detected person. When the place where the person is detected is the kitchen, the utterance control unit 1132 determines, for example, "what are you doing in the kitchen now?" As the utterance content.

When the identification reliability is medium, the process proceeds to step S153, and the utterance control unit 1132 determines the utterance content so as to confirm who the detecting person is, with the content not to be unnatural. The utterance control unit 1132 uses, for example, the content including the name of the identification result of the first identification unit 1131 as the utterance content. When the identification result of the first identification unit 1131 is "mother", the utterance control unit 1132 determines, for example, "Mother, what are you doing in the kitchen?" As the utterance content.

If the identification reliability is low, the process proceeds to step S154, and the utterance control unit 1132 sets the utterance content to be a content that more directly asks who the detection person is. The utterance control unit 1132 determines, for example, "Who is in the kitchen?" As the utterance content.

The text data of the utterance content determined in step S15 is passed from the utterance control unit 1132 to the voice output unit 114. In step S16, the voice output unit 114 converts the spoken text into voice data by voice synthesis and outputs it from the speaker 300. In step S17, the voice input unit 115 acquires the response to the system utterance from the microphone 400.

In step S18, the second identification unit 1133 performs personal identification based on the input voice. The second identification unit 1133 performs identification based on acoustic features (acoustic analysis) and discrimination based on language features (semantic analysis). Identification based on acoustic features can be obtained if a response voice is obtained, but discrimination based on language features makes it unclear who the question and response are. However, in identification based on language characteristics, the meaning such as "(I am) mother" can be taken into consideration, so the identification result is considered to be reliable.

In step S19, the person identification unit 1134 identifies the detected person in consideration of the image-based identification result (S13) by the first identification unit 1131 and the voice-based identification result (S18) by the second identification unit 1133. ..

If the utterance is "what are you doing in the kitchen?", It is assumed that you will get "cooking now" as a response. In this case, the second identification unit 1133 cannot perform identification based on language features, but can obtain identification results based on acoustic features. Here, if the identification result of the second identification unit 1133 matches the identification result of the first identification unit 1131, the person identification unit 1134 can confirm that the identification result of the first identification unit 1131 is correct, and this is finally confirmed. Make it a specific result. On the other hand, when these two identification results are different, the person identification unit 1134 may adopt one of the identification results, or the detected person may be unknown. In this case, the person identification unit 1134 may set the identification reliability low to more directly confirm who the detected person is in the next utterance.

If the utterance is "Mom, what are you doing in the kitchen?", And if the person is actually "Mother", it is assumed that "Cooking now" will be obtained as a response, and "Mother". If not, it is expected that you will get a response such as "I am not a mother" or "I am an older sister". In either case, the second identification unit 1133 can perform both identification based on acoustic features and discrimination based on linguistic features. In identification based on language characteristics, in response to a question using the name "mother", whether or not the other party is the "mother" is identified based on whether or not the response sentence contains a phrase that denies it. it can. If the response sentence contains a phrase indicating who you are, you can identify the person to be detected based on it. At this time, it is possible that the discrimination result based on the acoustic feature and the discrimination result based on the language feature may differ, but the discrimination result based on the language feature may be prioritized, and the judgment is made in consideration of the respective discrimination reliability. You may.

If the utterance is "Who is in the kitchen?", It is assumed that the response will be "I am XX". Therefore, the second identification unit 1133 may identify the detected person by performing identification based on the language feature. Since the utterance content that directly inquires who the detection person is is adopted, the detection person can be more reliably identified from the meaning content of the response. At this time as well, the second identification unit 1133 may perform identification based on the acoustic features in the same manner as described above.

In step S20, it is determined whether or not to continue feature collection. If the feature collection is to be continued, the process returns to step S10 to process the next frame.

When the feature collection is completed, in step S21, the feature registration unit 130 registers the acquired human body / biological feature in the storage unit (not shown) in association with the detection result of the detected person. In the flowchart of FIG. 3, the feature registration is performed at the timing when the feature collection is completed, but the feature registration may be performed at the timing when the tracking of the detected person is completed. The feature registration unit 130 registers all the features obtained from the start to the end of tracking in association with one person. However, the feature registration unit 130 may divide one tracking period into a plurality of parts and register the obtained features in association with the person whose identification result is within the period for each period. This process cannot be correctly recognized that the person has been replaced in the middle, and the human body detection unit 112 has detected it as the same person, but it can be performed when the person identification unit 113 identifies the person as a different person.

[Advantageous effect of this embodiment]
According to the present embodiment, personal identification based on an image and personal identification based on voice are performed, and the final identification result can be obtained by integrating both, so that accurate identification can be performed. In particular, when accurate identification cannot be performed only from an image, accurate identification is possible by making a system utterance, acquiring a voice response from the user, and performing identification based on voice. Further, by speaking only when the identification result based on the image is unreliable, or by determining the content of the utterance according to the identification reliability based on the image, it is possible to minimize annoyance to the user.

(Second Embodiment)
In the first embodiment, as shown in FIG. 4, the utterance content is determined according to the identification reliability. However, in the present embodiment, the content of the utterance is determined in consideration of the fact that if a voice response is obtained from the user, at least the identification based on the acoustic characteristics can be performed and the utterance can be made a plurality of times in one dialogue. The process is changed from the first embodiment. Hereinafter, the differences from the first embodiment will be mainly described.

FIG. 6 is a flowchart showing the flow of the utterance content determination process in the present embodiment. It should be noted that the present embodiment also assumes that the utterance is performed a plurality of times, and therefore the process shown in FIG. 6 is not the process itself of step S15 in FIG.

When it is time to make an utterance, the utterance control unit 1132 determines the utterance content that enables natural communication, as shown in step S31. Therefore, it is not necessary to include the name of the detected person in the utterance content, and the content such as "what are you doing in the kitchen now?" Is determined as the utterance content.

In step S32, the content determined in step S31 is output from the speaker 300, and as a result, the second identification unit 1133 identifies the user's response voice acquired by the microphone 400. Here, at least the identification based on the acoustic characteristics may be performed. Of course, identification based on language characteristics may be carried out if possible.

In step S33, it is determined whether the identification result by the first identification unit 1131 and the identification result by the second identification unit 1133 match. If they match, the person to be detected can be identified, so there is no need to speak further. On the other hand, when the two identification results are different, further utterance content is determined in step S34 in order to more reliably identify the person. The utterance control unit 1132 compares the content of the second utterance with the content of the first utterance, and determines more directly as the content for confirming who the detection person is. In this case, for example, the utterance content "Isn't it the mother in the kitchen?" Can be adopted. Alternatively, as in the case where the reliability of the first embodiment is medium or low level, the utterance content including the name of the detected person (eg, "Mom, what are you doing there?") And who is it. The content of the utterance (eg, "Who is in the kitchen?") May be determined as the content of the second utterance.

According to the present embodiment, the same effect as that of the first embodiment can be obtained, and more natural dialogue can be performed when performing identification based on voice.

In the above description, it is explained that the utterance is performed twice in one dialogue, but the utterance may be performed three times or more in one dialogue. In that case, the first few times may be a natural utterance, and when the person cannot be identified based on the voice, the utterance may be made to confirm who the detected person is.

Further, in the processing of the present embodiment, in the first embodiment, when the image-based identification reliability is high and the utterance of natural content is performed, the identification result based on the voice is different from the identification result based on the face map. It is also applicable in some cases.

(Other)
Each of the above embodiments is merely an example of the present invention. The present invention is not limited to the above-mentioned specific form, and various modifications can be made within the scope of its technical idea.

In the above description, the timing and contents of the feature collecting device 100 (domestic robot) talking to the user have been described, but this is a process for utterance for obtaining a voice-based response from the user. When implementing a home robot as a communication robot, it is not necessary to apply the above processing to utterances for communicating with the user. Further, in the above explanation, as an example of natural communication, an utterance that does not include the other party's name is given, but if the utterance including the name is a natural scene or the other party, the utterance content including the name is a natural utterance. It becomes.

In the above embodiment, the example in which the feature collecting device 100 is mounted on the home robot has been described, but it may be mounted on a surveillance camera or the like. Further, the personal identification device 110 does not need to be mounted on the feature collecting device 100 for collecting learning features, and may be used by itself to identify an individual.

(Additional note)
Audio output means (14,114) that output audio and
Voice input means (15,115) for acquiring voice in response to output voice, and
Image input means (11,111) and
Detection means (12,112) for detecting a person from a moving image input to the image input means, and
Person identification means (13,113) for identifying a person detected by the detection means, and
With
The person identification means
A first identification means (131, 1131) for identifying the person based on the moving image, and
It has a second identification means (133, 1133) for identifying the person based on the input voice obtained as a response to the output voice.
The person is identified based on the first identification result which is the identification result by the first identification means and the second identification result which is the identification result by the second identification means.
A personal identification device characterized by the fact that.

10: Personal identification device 11: Image input unit 12: Human body detection unit 13: Person identification unit 131: First identification unit 133: Second identification unit 14: Voice output unit 15: Voice input unit 100: Feature collection device 110: Individual Identification device 111: Image input unit 112: Human body detection unit 113: Person identification unit 1131: First identification unit 1131: Speech control unit 1133: Second identification unit 1134: Human body identification unit 114: Voice output unit 115: Voice input unit 120 : Feature acquisition unit 130: Feature registration unit 200: Camera 300: Speaker 400: Microphone

Claims (11)

Audio output means to output audio and
A voice input means for acquiring voice that responds to output voice,
Image input means and
A detection means for detecting a person from a moving image input to the image input means,
A person identification means for identifying a person detected by the detection means, and
With
The person identification means
A first identification means for identifying the person based on the moving image,
It has a second identification means for identifying the person based on the input voice obtained as a response to the output voice.
The person is identified based on the first identification result which is the identification result by the first identification means and the second identification result which is the identification result by the second identification means .
When the reliability of the first identification means is less than the first threshold value and greater than or equal to the second threshold value smaller than the first threshold value, the content including the name of the person as the first identification result is included as a call. , Determined as the output voice,
When the reliability by the first identification means is less than the second threshold value, the content of inquiring who the person is is determined as the content of the output voice.
A personal identification device characterized by the fact that. When the reliability of the first identification means is equal to or higher than the first threshold value, the person identification means uses the first identification result as the identification result of the person.
The personal identification device according to claim 1 . The output audio from the audio output means is output at a predetermined timing.
The predetermined timing is
The timing when the identification reliability of the person becomes less than the threshold value,
The timing at which the first predetermined time has elapsed from the timing at which the person was detected,
The timing at which the state in which there is almost no time change of the person continues for the second predetermined time,
Timing when the person goes out of the imaging range,
Timing the distance between the person and the personal identification device is equal to or less than a predetermined distance,
At least one of
The personal identification device according to claim 1 or 2 . When the first identification result and the second identification result do not match, the person identification means newly outputs an output voice and the second identification means is based on an input voice that responds to the new output voice. Identify by
The content of the new output voice is the content that confirms the person more directly than the content of the previous output voice.
The personal identification device according to any one of claims 1 to 3 . The second identification means identifies the person by performing at least one of waveform analysis and language analysis using the input voice.
The personal identification device according to any one of claims 1 to 4 . The first identifying means identifies the person based on at least one of facial features, human body features, and behavioral features obtained from the moving image.
The personal identification device according to any one of claims 1 to 5 . The personal identification device according to any one of claims 1 to 6 .
A feature acquisition means for acquiring at least one of the detected features related to the human body or behavior of the person from the moving image input to the image input means.
A feature registration means for registering a feature acquired by the feature acquisition means in association with a person identified by the person identification means, and a feature registration means.
A feature collection device equipped with. A personal identification method performed by a computer
A detection step that detects a person from a moving image,
The first identification step of identifying the person based on the moving image and
Audio output step to output audio and
A voice input step to get the voice that responds to the output voice,
A second identification step of identifying the person based on the input voice obtained in response to the output voice, and
A third identification step that identifies the person based on the first identification result, which is the identification result in the first identification step, and the second identification result, which is the identification result in the second identification step.
Only including,
In the audio output step,
When the reliability in the first identification step is less than the first threshold value and greater than or equal to the second threshold value smaller than the first threshold value, the content including the name of the person as the first identification result is included as a call. , Determined as the output voice,
When the reliability in the first identification step is less than the second threshold value, the content of inquiring who the person is is determined as the content of the output voice.
Personal identification method. When the reliability in the first identification step is lower than the first threshold value, performs the voice output step, the speech input step, the second identification step,
The personal identification method according to claim 8 . When the reliability in the first identification step is equal to or higher than the first threshold value, the first identification result is used as the identification result of the person in the third identification step.
The personal identification method according to claim 9 . A program for performing each step of the method according to any one of claims 8 to 10 .

Images