JP2006263348A - Device, method, and program for identifying user - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify a user with high precision through the use of a plurality of kinds of biological information by examining the identity of the user with the same tracking directivity. <P>SOLUTION: A user identifying device comprises: a speaker identifying part 104 for identifying the user from the first biological information acquired by a microphone and an all-sky camera; a face identifying part 105 for generating first living body identification information which is obtained by relating user identification information to a user existence direction, identifying the user, based on the biological information outputted from the all-sky camera, and generating second living body identification information which is obtained by relating the user identification information to the user existence direction; an area tracking part 106 for detecting the approximation area of an image feature area in an image in the direction indicated by the first living body identification information from a new input image, setting it as the image feature area, and obtaining the detection direction; and an identify examining part 110 for determining that the user by the first living body identification information is the same as the user by the second living body identification information when a difference between the detection direction of the image feature area and the direction of the second living body identification information acquired after the first living body identification information is not more than a prescribed threshold. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a user identification device, a user identification method, and a user identification method for identifying a user based on a user's biometric information obtained by detecting the user and a biometric dictionary in which biometric information for each user is registered. It relates to a user identification program.

  In recent years, various intercoexistence robot technologies that share an activity space with humans have been researched and developed. In such a human-symbiotic robot, it is an important issue to accurately identify and authenticate who the user is. With regard to such user identification, in particular, speaker identification technology used to identify a user's biometric information (biometrics) such as a user's voice and face that can be acquired from a relatively remote location, A user authentication method based on face identification technology is promising.

  The speaker identification method is as follows. A voice print pattern of an actual user's voice is acquired in advance, and a voice print dictionary in which the voice print pattern is associated with each user is generated. Then, the voice print pattern of the input voice and the voice print pattern registered in the voice print dictionary are collated to obtain a similarity indicating the degree of matching of the voice print patterns. Then, if there is a voice print pattern in which the similarity is the maximum above a predetermined threshold, the input voice is determined to be the voice of the user corresponding to the voice print pattern with the maximum similarity. is there.

  Similarly to the speaker identification method, the face identification method acquires an actual user's face pattern in advance, generates a face dictionary that associates the face pattern for each user, and inputs the face The face pattern of the image and the face pattern registered in the face dictionary are collated to determine the similarity, and if there is a face turn that maximizes the similarity above a predetermined threshold, the input face image is the maximum This is a method for determining that the face image of the user corresponds to the face pattern of the similarity.

  As a user identification technique using such user's biological information, there is a technique disclosed in Patent Document 1, for example. In the technique of Patent Document 1, in the personal authentication system, a plurality of types of biometric information such as a voice print pattern and a face pattern are input, and the similarity obtained by identifying each of the plurality of types of biometric information is further obtained. The final similarity for each input biometric information is determined from the series of similarities collected for each type of biometric information (voice print, face, etc.). Such determination is performed by selecting a maximum value of time-series similarity or calculating an average value. The personal authentication system authenticates the user by comprehensively determining the final similarity determined for each piece of biometric information.

  In the technique of this patent document 1, since the time-series similarity is used for user identification, a plurality of pieces of biological information that are input substantially simultaneously within a period in which the time-series similarity is output are: It is assumed that it is premised on the biometric information of the same user.

JP 2000-148985 A

  However, when such a short period of time is regarded as substantially simultaneous and it is determined that a plurality of pieces of biometric information input during this time are biometric information of the same user, there are the following problems. .

  That is, there is a timing in the biological information that can be acquired from the user. For example, the timing at which the voiceprint pattern can be acquired is limited to when the user utters, and the timing at which the face pattern can be acquired is limited to the timing at which the front of the user's face faces the direction of the imaging camera. As described above, a plurality of types of biological information cannot always be acquired almost simultaneously, and each biological information can be acquired only at different times, or only one biological information can be acquired.

  For this reason, if a certain short period is regarded as substantially simultaneous like the technique of patent document 1, and it will be judged that the multiple types of biological information acquired in the period is the same person's user, An error may occur in the verification of identity. For example, even if the face image and the sound can be acquired substantially simultaneously, even if the user who has generated the sound is different from the user of the acquired face image, the technique of Patent Document 1 may determine that the person is the same person. There is. On the other hand, when the face image and the sound are acquired after a period that is considered to be substantially the same, the user of the face image and the user who emitted the sound may be the same person, but only with approximately the same time. Since it is judged whether or not they are the same person, there is a possibility that they are judged as different users.

  By the way, in the user identification technique, a biometric dictionary (such as a voice print dictionary or a face dictionary) for user authentication is often generated when the user identification system starts to operate for a user. . The biological dictionary generated at this point in time is considered to have an expiration date in the period in which biological information can be used due to subsequent changes in the noise environment and lighting environment, and changes in the user's own voice and face over time. Good. That is, when the expiration date of the biometric information expires, the degree of similarity according to the biometric dictionary in which the biometric information is registered is not sufficiently increased even by the user himself, and the accuracy of user identification is reduced. For this reason, in general, dictionary maintenance is performed in which an administrator with system management authority relearns biometric information whose biometric information has expired by an explicit operation when the biometric information has expired. ing.

  In the user identification system having such a dictionary maintenance function, when the same person is authenticated by biometric information acquired almost simultaneously as in Patent Document 1, the following problem occurs. That is, while the face pattern of the user A is identified as that of the user A, the user tries to identify the voice uttered by the user B who is not registered in the biological dictionary on the side. If it cannot be specified, it is erroneously determined that the user A's voiceprint dictionary has expired based on this fact, and a completely different voiceprint pattern of the user B is used as the voice of the user A that can be identified by the face pattern. There is a danger of learning and updating the biological dictionary.

  The present invention has been made in view of the above, and can perform user identification using a plurality of types of biological information with high accuracy by performing verification of user identity based on tracking directionality. It is an object to provide a user identification device, a user identification method, and a user identification program.

  In order to solve the above-described problems and achieve the object, the present invention detects a user's voice and outputs the detected voice as user's biological information, and captures a user's image. A plurality of imaging means for outputting captured images as biometric information of a user, and a plurality of user identification information and biometric information of the user associated with each biometric information output by the imaging means and the detection means. A user is identified from the biometric information based on the biometric dictionary storing means for storing the biometric dictionary, the biometric information output by the detecting means, and the biometric dictionary corresponding to the biometric information, and identification of the user Biometric identification information associated with information and a direction in which the user exists is generated, and the user is identified from the biological information based on the biological information output by the imaging unit and the biological dictionary corresponding to the biological information. Separately, identification means for generating biometric identification information associated with the identification information of the user and the direction in which the user exists, and the direction indicated by the first biometric identification information generated by the identification means and acquired earlier An area that approximates an image feature area that is a characteristic area for an image is detected from an image newly input by the imaging unit, and the detected area is set as a new image feature area. Area tracking means for obtaining a new detection direction of the image feature area, and second biometric identification information acquired after the detection direction of the image feature area obtained by the area tracking means and the first biometric identification information It is determined whether or not the difference between the direction and the second direction is less than or equal to a predetermined threshold, and when the direction difference is less than or equal to the threshold, the user identified by the first biometric identification information and the second of And identity verification means the user identified by the body identification information is determined to be identical, a user identification device characterized by comprising a.

  Moreover, this invention is the user identification method and user identification program which are performed with the said user identification device.

  According to the present invention, the user is identified based on the biological information output by the detection means and the biological dictionary corresponding to the biological information, and the biological information is associated with the user identification information and the direction in which the user exists. On the other hand, identification information is generated. On the other hand, a user is identified based on biometric information output by a separate imaging unit and a biometric dictionary corresponding to the biometric information, and the user's identification information and the user exist. Biometric identification information associated with the direction is generated. That is, the direction in which the user exists is obtained separately from the biological information obtained from the detection means and the biological information obtained from the imaging means. And the area | region which approximates the image feature area which is a characteristic area | region regarding the image of the direction which the 1st biometric identification information acquired previously is tracked is tracked after the detection direction of the image feature area and the first biometric identification information. It is determined whether or not a difference from the direction of the acquired second biometric identification information is equal to or smaller than a predetermined threshold value. If the difference in direction is equal to or smaller than the threshold value, the first biometric identification information is identified. Since it is determined that the user and the user identified by the second biometric identification information are the same, when performing user identification using a plurality of types of biometric information, they are obtained from separate biometric information. The identity of the user can be verified based on the identity of the user's direction, and even when different biological information is acquired from the user after a certain period of time, the identity of the user can be verified with high accuracy. Can be done.

  Exemplary embodiments of a user identification device, a user identification method, and a user identification program according to the present invention are explained in detail below with reference to the accompanying drawings. In the present embodiment, the user identification device of the present invention is applied to an autonomous mobile robot.

(Configuration of autonomous mobile robot)
FIG. 1 is a block diagram showing a functional configuration of the autonomous mobile robot according to the present embodiment.

  As shown in FIG. 1, the autonomous mobile robot according to the present embodiment includes three microphones 206, 207, 208, an all-sky camera 205, an acoustic directivity forming unit 101, a vocabulary identifying unit 103, and a speech. Person identification unit 104, face identification unit 105, area tracking unit 106, user authentication unit 107, dictionary update unit 108, service provision unit 109, identity verification unit 110, and biological dictionary storage unit 120 It has a configuration mainly equipped with. Note that the autonomous mobile robot according to the present embodiment includes a part that performs functions other than the user identification function, such as a control unit that performs movement control, in addition to the units illustrated in FIG.

  Here, the microphones 206, 207, and 208 correspond to the detection means in the present invention, and the all-sky camera 205 corresponds to the imaging means in the present invention. The speaker identification unit 104 and the face identification unit 105 correspond to the identification unit in the present invention.

  Microphones 206, 207, and 208 are for inputting user's voice. The omnidirectional camera 205 is a camera with a fisheye lens with an upward visual field of 180 degrees, and can image 360 degrees around the robot body including the user.

  Here, FIG. 2 is a schematic diagram showing an appearance of the autonomous mobile robot according to the present embodiment. As shown in FIG. 2, the autonomous mobile robot according to the present embodiment includes a main body 201, driving wheels 202 and 203 and auxiliary wheels (not shown) for movement mounted on the main body 201, and is used. It is configured to be able to move freely on the floor in the person's residence.

  A pair of left and right microphones 206 and 207 described above are horizontally disposed on the front surface of the main body 201, and the microphone 208 is disposed on the back surface of the main body 101 so as to be positioned in substantially the same horizontal plane as the microphones 206 and 207. . The directivity of the user's voice input by the three microphones 206, 207, and 208 that are spatially discretely arranged in this manner is obtained by a plurality of acoustic directivity forming modules 102 of the acoustic directivity forming unit 101 described later. Multiple settings can be made in the horizontal and vertical directions.

  The above-described all-sky camera 205 is attached to the upper part of the main body 201 of the autonomous mobile robot. The all-sky camera 205 is a fisheye lens-equipped camera 315 having a visual field of 180 degrees upward, and can acquire a full-sky image 318 having a circular field of view. The all-sky camera 205 images the surroundings at fixed time intervals and outputs captured image frames.

  FIG. 3A is a schematic diagram illustrating an arrangement of the omnidirectional camera 205 that captures images of two users. FIG. 3B is a celestial camera showing two users arranged as illustrated in FIG. FIG. 6 is a schematic diagram illustrating a whole sky image captured at 205. When the users 316 and 317 captured by the omnidirectional camera 205 are at the positions shown in FIG. 3A, the two users 316 and 317 are projected in the omnidirectional image 318 as shown in FIG. 3-2. Is done.

  Returning to FIG. 1, the acoustic directivity forming unit 101 separates and extracts voices coming from various directions around the autonomous mobile robot for each direction, and obtains directional voice information composed of voice data of the voice section and the arrival direction. A plurality of acoustic directivity forming modules 102 that individually and selectively pass sound coming from various surrounding directions, and a directivity setting value storage unit (not shown) that stores the arrival directions; It has. Voices separated and extracted by the acoustic directivity forming unit 101 include user voices and environmental noises.

  The sound directivity forming unit 101 evaluates the strength of each of the separated and extracted voice information, and after the strength falls below the second threshold from the time that has passed a predetermined period after the strength becomes equal to or greater than the first threshold. The time until a predetermined time elapses is defined as a “voice section”, and data obtained by extracting only the voice section from the voice information is set as voice data of the voice section. Then, the direction information indicating the arrival direction and the voice data of the voice section are associated with each other and output as directional voice information. It should be noted that the voice section detection process based on intensity has the effect of excluding voices that contain only weak environmental noise coming from a distance so as not to be processed in a later stage.

  FIG. 4 is a block diagram illustrating a configuration of the sound directivity forming unit 101. The sound directivity forming unit 101 inputs the front beamformers 410, 420, and 430 for inputting the sound from the microphones 206 and 207, the sound from the microphone 208, and the output from the previous beamformers 410, 420, and 430. , 421, 422, 423,... 431, 432, 433,..., And a directivity setting value storage unit 440. Here, as shown in FIG. 4, the subsequent beamformers 411, 412, 413,... Receive the output from the preceding beamformer 410 and the sound from the microphone 208, and the subsequent beamformers 421, 422. , 423,... Input the output from the beam former 420 and the sound from the microphone 208, and the beam formers 431, 432, 433,. The sound from the microphone 208 is input.

  Each of the subsequent beamformers 411, 412,... By the directivity setting value storage unit 440, the preceding beamformer 410, and the subsequent beamformers 411, 412, 413,. The number of acoustic directivity forming modules 102 corresponding to 413,. The same applies to the subsequent beamformers 421, 422, 423,... Corresponding to the preceding beamformer 420, and the subsequent beamformers 431, 432, 433,.

  In other words, the acoustic directivity forming module 102 corresponds to one area with the beam former 410, the beam former 411, and the directivity storage unit 440 in the preceding stage, and the acoustic directivity forming module 102 corresponding to another area corresponds to the acoustic directivity forming module 102. The former beamformer 410, the latter beamformer 412, and the directivity storage unit 440.

  FIG. 5 is a block diagram showing a configuration of the sound directivity forming module 102. In FIG. 5, the example comprised by the beam former 410 of the front | former stage and the beam former 411 of the back | latter stage is shown. The acoustic directivity forming module 102 has a function of freely setting the directivity of audio input from the three microphones 206, 207, and 208 in the horizontal and vertical directions.

  The beam former 410 in the previous stage inputs the sound from the two microphones 206 and 207 arranged horizontally on the front surface of the autonomous mobile robot, and sets the angle θ set with respect to the line segment connecting the microphone 206 and the microphone 207. The directivity forming means has a function of suppressing sensitivity in a direction deviating from the central directivity direction as the central directivity direction. As a result, the sensitivity with respect to the predetermined range centered on the central directivity direction is maintained at a predetermined value or higher, and the sensitivity in the other directions is suppressed to less than the predetermined value. Various configurations of such beamformers have been proposed. For example, in a method of calculating a delay sum of a plurality of microphone inputs known as a phased array, directivity is directed in an arbitrary direction by controlling the delay amount. be able to. There is also known a technique that uses two Griffith-Jim type generalized sidelobe cancellers to more clearly extract speech within a set directivity range.

  As shown in FIG. 6, the center directivity direction formed by the two microphones 206 and 207 is a conical surface 612 opened by an angle θ from the line segment connecting the microphone 206 and the microphone 207 as illustrated. Therefore, the output is mainly composed of sound from a sound source that exists within a predetermined range near the conical surface 621.

  However, since the directivity range formed by the beamformer 410 covers the entire vicinity of the conical surface 621, the directivity range is too wide to know from which direction of the range the sound has come. Therefore, the second beamformer 411 is connected to the subsequent stage of the beamformer 410. The beamformer 411 is a sidelobe canceller that inputs the output of the beamformer 410 and the sound from the microphone 208 disposed on the back of the autonomous mobile robot and forms a predetermined directivity range.

  As described above, the beam former 411 is provided at the subsequent stage of the beam former 410, and by inputting the output from the beam former 410 and the sound from the microphone 208, a virtual position located between the two microphones 206 and 207 is obtained. The sound from the microphone 501 and the sound from the back microphone 208 are input, and the center of directivity is a conical surface opened by a predetermined angle from the line segment connecting the virtual microphone 501 and the back microphone 208. Become. At this time, since the directivity range of the sound from the virtual microphone 501 is already narrowed by the beam former 410, the output of the beam former 411 connects the conical surface 621, the virtual microphone 501 and the rear microphone 208. The sound from the intersecting direction of the conical surfaces opened by a predetermined angle from the line segment becomes the strongest.

  In order to obtain such an action, in the autonomous mobile robot of the present embodiment, three microphones 206, 207, 208 are arranged so as to form a horizontal isosceles triangle with the microphone 208 as a vertex. Yes.

  FIGS. 7A and 7B are schematic diagrams illustrating directivity ranges set by the former beamformer and the latter beamformer of the acoustic directivity module. FIG. 7-1 is a diagram viewed from above the autonomous mobile robot, and FIG. 7-2 is a diagram viewed from the horizontal direction. Reference numeral 74 denotes a front direction of the main body 201 of the robot.

  The conical surface 70 (621) by the beamformer 410 represents the horizontal directionality of the directivity at the height of the microphone, and the conical surface 71 by the subsequent beamformer 411 has an effect of further narrowing the horizontal directionality of the directivity in the vertical direction. Have. For this reason, the directivity range by the former beamformer 410 is called the horizontal directivity range 70, and the directivity range by the latter beamformer 411 is called the vertical directivity range 71.

  The directivity range 72 by the beamformer 410 and the beamformer 411 appears as a region 72 sandwiched between two cones of a horizontal directivity range 70 and a vertical directivity range 71 as shown in FIGS.

  That is, the directivity range in the vertical plane by the beam former 410 is also a region sandwiched between two cones starting from the position of the microphone height 73 shown in FIG. At this time, the common region 72 of the horizontal directivity range 70 and the vertical directivity range 71 is the final directivity range of the acoustic directivity forming module. The sound directivity forming module passes sound coming from within the directivity range 72 with high sensitivity. When the distance of the sound source is sufficiently large with respect to the distance between the microphones 206, 207, and 208, the starting point of the formed directivity range, that is, the apex of the cone is included in the line segment connecting the virtual name microphone 501 and the microphone 208. It is good also as a point.

  Each azimuth angle of the horizontal directivity range 70 and the vertical directivity range 71 is set so as to open, for example, 20 degrees. The beam former 410 and the beam former 411 can thus form a narrow directivity range of, for example, ± 10 degrees (that is, an angle of 20 degrees).

  The beamformer 410 and the beamformer 411 set the directivity range, that is, the opening angle of the horizontal directivity range 70 and the vertical directivity range and the angle width through which sound can pass, in the directivity range setting value storage unit 440. Read / write is possible from the outside.

  FIG. 8A is an explanatory diagram illustrating an arrangement example of the directivity range that covers 180 degrees in increments of 20 degrees. In the autonomous mobile robot of this embodiment. By preparing nine directivity ranges formed in this way and arranging them in tiles as shown in FIG. 8A, the range of ± 90 degrees (ie 180 degrees) is divided into nine directivity ranges. .

  In this way, a narrow directivity range can be set at an arbitrary position on the spherical dome centering on the autonomous mobile robot by the single acoustic directivity forming module 102. In the present embodiment, a plurality of tiles that cover a predetermined range in the horizontal and vertical directions are provided with a plurality of the sound directivity forming modules 102 and the directivity ranges of the sound directivity forming modules 102 are different for each module. By arranging in this way, the input voice is separated and extracted as the output of the module having the corresponding directivity range regardless of the direction from which the voice comes. Therefore, the sound directivity forming unit 101 operates all the sound directivity forming modules 102 at the same time, so that even when voices simultaneously arrive from different directions, they can be separated and output individually.

  For example, as shown in FIG. 8A, a maximum of N = 9 × 9 = 81 acoustic directivity forming modules 102 are prepared as combinations of nine horizontal and vertical ranges in 20 degree increments. A range of 360 degrees horizontally and 180 degrees around the robot can be covered. However, in practice, the number of effective combinations is less than 81 except for a range where the horizontal directivity range and the vertical directivity range do not intersect among the 81 combinations. In addition, since a plurality of vertical directivity ranges are set for one horizontal directivity range, the output of the former beamformer 410 is shared by the plurality of subsequent beamformers 411, 412, 413,. And the number of beamformers can be less than twice the number of valid combinations. The configuration of the acoustic directivity forming unit 101 shown in FIG. 4 corresponds to an effective combination, and a necessary number of beamformers that are partially shared are mounted.

  The outputs of the microphones 206 and 207 are output to the former beamformer 410 that realizes horizontal directivity. In order to divide the horizontal 180 degrees into 9 parts, the beam formers in the preceding stage are set with different directivity ranges, and a total of 9 beamformers are prepared. In the figure, three beamformers 410, 420, and 430 are shown as a part thereof.

  The outputs of the former beamformers 410, 420, and 430 that realize different horizontal directivity ranges are supplied to a plurality of subsequent beamformers that realize different vertical directivities. For example, the output of the beamformer 410 is output to the subsequent beamformers 411, 412, 413,. In order to divide the vertical 180 degrees into nine, a maximum of nine subsequent beamformers that commonly receive the output from one previous beamformer are required.

  Here, the autonomous mobile robot according to the present embodiment can move and turn on the floor. At this time, the directivity range of the acoustic directivity forming module 102 is set as a relative direction based on the microphones 206, 207, 208, that is, the robot body 201. When the floor is horizontal, the vertical component of the directivity range always follows a certain standard, but the standard of the horizontal component changes as the main body 201 turns.

  FIG. 8-2 and FIG. 8-3 are explanatory diagrams illustrating correction from the relative direction to the absolute direction in the autonomous mobile robot.

  For example, as shown in FIG. 8A, when the front direction of the main body 201 is directed in the direction of the arrow 840, the horizontal direction from the direction and the right direction is positively + θa, that is, the direction indicated by the arrow 841. Assume that the voice of the user 842 is detected. Next, as shown in FIG. 8B, if the main body 201 turns rightward by + φ and the front faces in the direction of the arrow 843, the voice of the user 842 is now heard from the front of the main body 201. Is observed in the direction of −θb. As is clear from this example, the voice of the same user 842 is detected in different directions as the main body 201 turns.

  Therefore, the horizontal azimuth offset angle θo is set to 0 with reference to the posture of the main body 201 in the state shown in FIG. In the state shown in FIG. 8B, since the main body 201 has turned to the right by the angle φ, the horizontal azimuth offset angle θo at this time is set to + φ with the right direction being positive. The horizontal azimuth θb of the user's voice in the state of FIG. 8-2 is corrected to an absolute azimuth based on the state of FIG. 8-1 using the horizontal azimuth offset angle θo. Assuming that the corrected θb is θb ′, θb ′ is calculated by the following equation.

θb '= θo−θb
As long as the user 842 does not move, the corrected θb ′ has the same value as θa regardless of the turning of the main body 201. The horizontal component of the direction information included in the directional sound information output from the acoustic directivity forming unit 101 is subjected to the above correction and converted into an absolute direction. As a result, even when the robot turns, the speaker identification information, face identification information, and area tracking information obtained in different postures can be compared with a common reference.

  Each acoustic directivity forming module 102 is assigned a region that is a directivity range for each module, and each region is assigned a region number for identifying the region. That is, the output from each acoustic directivity forming module 102 indicates that the voice data has arrived from the area to which the module is assigned. For this reason, each acoustic directivity forming module 102 uses direction information obtained by converting its module number into an area number assigned by a direction association table, which will be described later, and the direction information is added to the voice data of the input voice section. Sexual audio information is output. FIG. 9A is a data structure diagram of directional sound information output from the sound directivity forming module 102 of the sound directivity forming unit 101. The vocabulary identifying unit 103 and the speaker identifying unit 104 that input the directional speech information acquire the direction of arrival of the input speech by acquiring the direction information (region number) set in the directional speech information. can do.

  Returning to FIG. 1, the biological dictionary storage unit 120 is a storage medium such as an HDD (Hard Disk Drive Device), and stores a voiceprint dictionary 121, a face dictionary 122, and a vocabulary dictionary 123.

  For each user ID for identifying a user, the voiceprint dictionary 121 registers a user ID and a basic voice voice pattern such as “Aiueo” corresponding to the user ID in association with each other. It is a dictionary data file.

  The face dictionary 122 is a dictionary data file in which a user ID and a face image pattern obtained by actually capturing a user corresponding to the user ID are registered in association with each other for each user ID.

  The vocabulary dictionary 123 is a dictionary data file in which, for each user ID, a user ID, voice data, and a vocabulary indicating utterance contents corresponding to the voice data are associated and registered.

  The vocabulary identifying unit 103 collates the speech data set in the directional speech information output by the acoustic directivity forming unit 101 with the vocabulary dictionary 123 to identify the vocabulary that is the utterance content of the directional speech information. is there. Specifically, the vocabulary identification unit 103 compares the voice data set in the directional voice information with the voice data for each user ID registered in the vocabulary dictionary 123 to indicate the degree of matching of the voice data. The similarity is calculated, and the vocabulary string having the maximum similarity is used as the vocabulary identification result. And the vocabulary identification information which consists of a vocabulary string and direction information is output by making the arrival direction set to directional audio | voice information into direction information.

  FIG. 9-2 is a data structure diagram of vocabulary identification information. In the vocabulary identification information, when the speaker is a user registered in the vocabulary dictionary 123, as shown in FIG. 9-2, a vocabulary string having a meaning as a command to the autonomous mobile robot is a vocabulary identification. Set as a result. On the other hand, if the voice arriving at the autonomous mobile robot is, for example, voice other than a person or voice data of a user who is not registered in the vocabulary dictionary 123, vocabulary identification cannot be performed. The unit 103 sets the vocabulary unknown ID in the vocabulary identification information instead of the vocabulary string as the vocabulary identification result.

  The speaker identifying unit 104 is for identifying the speaker of the directional speech information by collating the speech data of the directional speech information output by the acoustic directivity forming unit 101 with the voiceprint pattern registered in the voiceprint dictionary 121. is there. Specifically, the speaker identification unit 104 compares the voice data set in the directional voice information with the voice print pattern for each user ID registered in the voice print dictionary 121 and matches the voice data with the voice print pattern. The degree of similarity indicating the degree of the above is calculated for each user ID, the user IDs having the highest similarity to the user IDs from the top N are selected, and the selected user IDs up to the top N are listed. Let the similarity column be the speaker identification result. Then, using the direction of arrival set in the directional speech information as direction information, speaker identification information composed of speaker identification results (up to the top N ranks of user ID and similarity) and direction information is output. .

  FIG. 9-3 is a data structure diagram of speaker identification information. When the voice data is a voice print pattern of a user ID registered in the voice print dictionary 121, as shown in FIG. 9-3, a column of user ID and similarity is set in the speaker identification result. However, speaker identification cannot be performed if the voice data is not of the user ID registered in the voiceprint dictionary 121 or if it is not a person's voice data. For this reason, the speaker identification unit 104 sets the unknown person ID as the speaker identification information instead of the column of the user ID and the similarity as the speaker identification result.

  The face identification unit 105 inputs image frames picked up by the omnidirectional camera 205 at regular intervals, and searches for face images searched in the input image frames and face images registered in the face dictionary 122. The user's face is identified in an image frame imaged by collating with a pattern. Specifically, the face identification unit 105 searches the face area from the input image frame, and collates the image data of the face area searched for with the face image pattern for each user ID registered in the face dictionary 122. The similarity indicating the degree of coincidence with the face image pattern of the image data is calculated for each user ID, and the user ID having the highest similarity to the user ID from the top N to the top N is selected and selected. The user IDs up to the top N and their similarity columns are used as face identification results. Then, direction information (area number) corresponding to the image center position of the face area is obtained by referring to a direction association table described later, and the face identification result (column up to the top N ranks of user ID and similarity) and direction The face identification information consisting of the information is output.

  FIG. 9-4 is a data structure diagram of face identification information. When the image data of the face area is a face image pattern of the user ID registered in the face dictionary 122, the face identification result includes a column of user ID and similarity as shown in FIG. 9-4. Although it is set, face identification cannot be performed when the image data of the face area is not the face image pattern of the user ID registered in the face dictionary 122 or the image data of the face of the person. Therefore, the face identification unit 105 sets the unknown person ID as the face identification information instead of the user ID and similarity column as the face identification result.

  FIG. 9-5 is a data structure diagram of a direction association table. The direction association table is stored in a storage medium such as a memory or an HDD, and is a table in which an area number, an acoustic directivity forming module number, and a coordinate range of an image center position of the area are associated with each other. The area number indicates the direction in which the user exists, and is set in the direction information in the above-described directional speech information, vocabulary identification information, speaker identification information, and face identification information.

  The acoustic directivity forming unit 101 estimates the arrival direction of the input speech. Since the acoustic directivity forming module 102 that estimated the arrival direction indicates the arrival direction as it is, in the direction association table, the region number and the acoustic directivity forming module are used. 102 are associated with each other.

  The face identification unit 105 obtains the image center position of the face area when obtaining the direction of the face area. The area number corresponding to the coordinate range including the coordinates of the image center position of the face area is the face area. Direction, and is set in the direction information of the face identification information.

  The area tracking unit 106 detects an image feature area, which is an area having set image characteristics, from each image frame input from the omnidirectional camera 205, and the image feature area is input for each input image frame. To track. The setting of the image feature area is performed by an area addition command output from the tracking direction verification unit 112 of the identity verification unit 110 described later. Further, the area tracking unit 106 outputs, as area tracking information, the detection tracking state (the current direction of the image feature area being tracked or the fact that the image feature area being tracked was lost) for each image feature area being tracked. To do.

  The identity verification unit 110 determines whether the user who is the target of the speaker identification information and the user who is the target of the face identification information are the same person, and is the same person The same set information in which the speaker identification information and the face identification information to be determined are set is generated, and the same direction verification unit 111 and the tracking directionality verification unit 112 are provided.

  The same direction verification unit 111 registers the direction information set in the biometric identification information acquired earlier in the speaker identification information and the face identification information, and acquires it after the direction information and the previous biometric identification information. It is determined whether or not the difference from the direction information set in the biometric identification information is equal to or less than a predetermined threshold value. The same direction verification unit 111 further determines that the user identified by the speaker identification information and the user identified by the face identification information are the same when the difference in the direction information is equal to or smaller than the threshold. Thus, the same set information is generated by combining the speaker identification information, the voice data thereof, the face identification information, and the image data thereof, and further adding an expiration date that determines the effective period of both identification information.

  The tracking co-direction verification unit 112 receives the region tracking information output from the region tracking unit 106 and determines whether the image feature region included in the region tracking information is detected, whether the speaker identification information and the face identification information are present. It is determined whether or not the difference from the direction information set in the biometric identification information acquired later is equal to or less than a predetermined threshold value. The tracking co-direction verification unit 112 further determines that the user of the speaker identification information and the user of the face identification information are the same person when the difference between the directions is equal to or less than the threshold, The same set information is generated by combining the information, its voice data, face identification information, and its image data.

  FIG. 9-6 is a data structure diagram of the same set information. FIG. 9-6 illustrates an example of the same set information generated by the same direction verification unit 111 and set with an expiration date. The same set information generated by the tracking same direction verification unit 112 is different from the same set information generated by the same direction verification unit 111 only in that this expiration date item is not set. The same applies to. As shown in FIG. 9-6, when the vocabulary identification information is output by the vocabulary identification unit 103, the vocabulary identification information is also set to the same set information. In FIG. 9-6, the speaker identification information and the face identification information are set as one set, but the same set information in which only one of the identification information is set may be output.

  The dictionary updating unit 108 analyzes the same set information generated by the identity verifying unit 110 and determines the necessity of updating each of the voiceprint dictionary 121 and the face dictionary 122 stored in the biological dictionary storage unit 120. In addition, there is a need to update the vital dictionary. Specifically, the dictionary update unit 108 performs the following dictionary update when both speaker identification information and face identification information exist in the same set information.

  That is, when the user ID having the maximum similarity in the speaker identification information and the user ID having the maximum similarity in the face identification information do not match, the user ID having the larger maximum similarity is selected as the user ID of this user. The bio-dictionary with the smaller similarity is updated. Further, when the unknown person ID is set for only one of the speaker identification information and the face identification information, the user ID of the identification information for which no unknown person ID is set is determined to be the user ID of this user. Then, the biological dictionary corresponding to the identification information in which the person unknown ID is set is updated.

  The user authentication unit 107 authenticates an actual user from the same set information generated by the identity verification unit 110. In the same set information, there may be information in which only speaker identification information is set, information in which only face identification information is set, or information in which both identification information is set.

  In addition, there are two types of speaker identification information: a user ID / similarity column and a person unknown ID, and face identification information also includes two types: a user ID / similarity column and a person unknown ID. In order to provide a service by authenticating a user, it is not necessary to consider the case where the user could not be identified by voice or face. For this reason, the user authentication unit 107 targets the maximum similarity between the speaker identification result and the face identification result only for the same set information that can identify the user by at least one of voice and face. Is output as user authentication information.

  When vocabulary identification information is set for the same set information, the vocabulary string set as the vocabulary identification result is also added to the user authentication information and output.

  The service providing unit 109 provides a predetermined service to the user based on the user authentication information output by the user authentication unit 108. As the service to be provided, an arbitrary service can be executed, for example, an e-mail check service. Since the user authentication information includes the user ID of the user specified by the user authentication unit 107, a service using this user ID can be provided. In addition, when the user authentication information includes a vocabulary string, it is possible to provide a service using the user ID as a command indicating the utterance content of the user's voice in the vocabulary string. For example, when configured to perform a mail check service, a service providing process is performed such as determining a mail check command from the vocabulary string and confirming the user's mailbox corresponding to the user ID. It is possible. On the other hand, in the case of user authentication information that does not include vocabulary identification information, the service providing unit 109 may be configured to return some result to the user as part of a previously ordered service.

(Overall processing of user identification)
Next, user identification processing by the autonomous mobile robot according to the present embodiment configured as described above will be described. FIG. 10 is a flowchart showing a procedure of overall user identification processing by the autonomous mobile robot according to the present embodiment.

  First, sound is input through the microphones 206, 207, and 208, and the sound directivity forming unit 101 generates directional sound information including sound data and direction information indicating the arrival direction (step S1001). The vocabulary identification unit 103 performs vocabulary identification processing based on the directional speech information and the vocabulary dictionary 123, and outputs vocabulary identification information including the vocabulary identification result and direction information (step S1002). Further, the speaker identification unit 104 performs speaker identification processing based on the directional audio information and the voiceprint dictionary 121, and outputs speaker identification information including the speaker identification result and direction information (step S1003).

  On the other hand, in parallel with this, the omnidirectional camera 205 images the surroundings at regular intervals and outputs an image frame (step S1004). The face identification unit 105 performs face identification processing based on the input image frame and the face dictionary 122, and outputs face identification information including the face identification result and direction information (step S1005).

  Next, the image feature region specified by the region addition command from the identity verification unit 110 is tracked for each image frame input by the region tracking unit 106, and region tracking information indicating the tracking state is output (step). S1006).

  When there is no movement of the user, the same direction verification processing is performed by the same direction verification unit 111 of the identity verification unit 110, and the speaker identification information verified with the same person and the face identification information are combined. The same set information is output (step S1007). On the other hand, when the user is moving, a region addition command is sent to the region tracking unit 106 by the tracking and directionality verification unit 112 of the identity verification unit 110, and the resulting region tracking information is transmitted to the region tracking unit. A tracking unidirectional verification process is received from 106 and the same set information combining the speaker identification information verified with the same person and the face identification information is output (step S1008).

  Next, user authentication processing is performed by the user authentication unit 107 to determine who is the user in the same set information, and user authentication information including the specified user ID is output (step S1009). ). Then, for the user specified by the user authentication information by the service providing unit 109, for example, a service providing process such as a mail check service is performed (step S1010). On the other hand, the dictionary update unit 108 updates the biological dictionary that needs to be updated from the same set information, and strengthens the biological dictionary (step S1011).

(Directed voice information generation processing)
Hereinafter, each process described in the overall process will be described in detail. First, generation processing of directional audio information by the acoustic directivity forming unit 101 will be described. FIG. 11 is a flowchart illustrating a procedure of directivity audio information generation processing by the acoustic directivity forming unit 101.

  First, when the microphones 206, 207, and 208 input sound (step S 1101), the acoustic directivity forming unit 101 periodically generates audio electrical signals of input sounds output from the microphones 206, 207, and 208 (A-D). (Analog-Digital) conversion (step S1102), and instantaneous amplitude values of digital audio signals obtained by the A-D conversion are sequentially input and accumulated in a FIFO (First-Input First-Out) (step S1103). .

  Of the amplitude value data stored in the FIFO, the amplitude value data from the microphones 206 and 207 are input to the beam formers 410, 420, and 430 in the previous stage and output as amplitude value data having a predetermined directivity. A first directivity formation process is performed (step S1104). Thereby, a horizontal directivity range is first obtained.

  .., 421, 422, 423... 431, 432, 433..., 431, 432, 433. Amplitude value data from the microphone 208 is input, and a second directivity forming process is performed by the subsequent beamformers 411, 412, 413,... S1105). As a result, a vertical directivity range is obtained, and a common range of the horizontal directivity range and the vertical directivity range obtained in the first directivity forming process becomes the directivity range of the input sound.

  Then, the acoustic directivity forming unit 101 sets the amplitude value data having the maximum amplitude value data output from each beamformer in the second stage in the second directivity forming process as the audio data of the audio section, and sets the maximum amplitude value data to The module number of the acoustic directivity forming module having the subsequent beamformer to be output is converted to the area number of the directivity range with reference to the direction association table to obtain direction information. Then, the acoustic directivity forming unit 101 generates and outputs directional voice information composed of voice data and direction information of this voice section (step S1106).

  Thus, in the directional audio information output from each acoustic directivity forming module 102 of the acoustic directivity forming unit 101, in addition to the audio data, direction information representing the directivity range of the acoustic directivity forming module 102 is included. Therefore, the vocabulary identifying unit 103 that inputs and uses directional speech information and the speaker identification 104 can easily determine the arrival direction of speech data from the input directional speech information.

(Vocabulary identification processing)
Next, vocabulary identification processing by the vocabulary identification unit 103 will be described. FIG. 12 is a flowchart showing a procedure of vocabulary identification processing by the vocabulary identification unit 103.

  First, the vocabulary identification unit 103 inputs new directional speech information output by the acoustic directivity forming unit 101 (step S1201). Then, the speech data set in the input directional speech information and the vocabulary string indicating the utterance content for each user registered in the vocabulary dictionary 123 are collated, and the similarity between the speech data and each vocabulary string is determined. The vocabulary string that is obtained and has the maximum similarity is acquired (step S1202).

  Next, it is determined whether or not the maximum similarity is greater than or equal to a predetermined threshold (step S1203). If the maximum similarity is greater than or equal to the threshold (step S1203: Yes), this vocabulary string is recognized as a vocabulary identification result, and the vocabulary identification consisting of this vocabulary string and the direction information set in the directional speech information. Information is generated and output (step S1204).

  On the other hand, if the maximum similarity is smaller than the threshold value in step S1203 (step S1203: No), the vocabulary unknown ID is used as the vocabulary identification result, and the vocabulary identification result and the direction information set in the directional speech information are included. Vocabulary identification information is generated and output (step S1205).

  Note that this vocabulary identification information can be used as a service-related command during service provision processing in the service provision unit 109.

(Speaker identification processing)
Next, speaker identification processing by the speaker identification unit 104 will be described. FIG. 13 is a flowchart showing a procedure of speaker identification processing by the speaker identification unit 104.

  First, the speaker identifying unit 104 inputs new directional speech information output by the acoustic directivity forming unit 101 (step S1301). Then, the audio data set in the input directional audio information is subjected to DFT (Discrete Fourier Transform) processing with a frame length F and a frame interval D, and a short time consisting of F / 2 elements. The power spectrum is converted into audio data arranged in time series (step S1302).

  Next, the voice data converted into the power spectrum sequence is compared with the voice print pattern for each user ID registered in the speaker dictionary 121, and the voice data and the voice print pattern of the user ID are compared for each user ID. A degree of similarity indicating the degree of coincidence is obtained, and a list of user IDs up to the top N ranks (N is an arbitrary integer) is obtained (step S1303).

  Next, it is determined whether or not the maximum similarity in the list of user IDs is equal to or greater than a predetermined threshold (step S1304). If the maximum similarity is equal to or greater than the threshold (step S1304: Yes), a column composed of the user IDs up to the top N and the similarities corresponding to the user IDs is recognized as a speaker identification result. Then, speaker identification information composed of a column composed of each user ID and each similarity and direction information set in the directional speech information is generated and output (step S1305).

  On the other hand, if the maximum similarity is smaller than the predetermined threshold in step S1304 (step S1304: No), the unknown person ID is set as the speaker identification result, and the direction set in the speaker identification result and the directional voice information is set. Speaker identification information composed of the information is generated and output (step S1306).

  Here, the collation between the voice data and the voiceprint dictionary 121 performed in step S1303 will be described. FIG. 14 is an explanatory diagram showing an example of voice data converted from input voice data into a power spectrum sequence. 1401, shows a voiceprint pattern of male A that say "Hello app kun". As can be seen from FIG. 14, the majority of the human voice has a harmonic structure composed of the fundamental frequency and its harmonics. And most of the period of the harmonic structure is composed of the power spectrum of vowels, reflecting the individuality.

  15A is an explanatory diagram showing an example of a voice print pattern 1402 when male A utters “Aiueo”, and FIG. 15-2 shows a voice print pattern 1403 when female B utters “Aiueo”. It is explanatory drawing which shows an example. As can be seen from both figures, the interval between the harmonic structures is larger in women than in men, that is, the fundamental frequency is higher.

  Thus, since most periods of the harmonic structure are composed of the vowel power spectrum and reflect the individuality, the voiceprint dictionary 121 stored in the biological dictionary storage unit 120 includes the voiceprint dictionary 121 shown in FIGS. A voice print pattern when a vowel as shown by -2 is uttered is registered for each user (for each user ID).

  The voiceprint dictionary 121 is generated based on a set of short-time power spectra obtained in a time series from teaching voice data in which each user utters all vowels of the language. In the case of Japanese, the voice data for teaching is voice data uttered “Aiueo” including five vowels. The power spectrum S (t) at time t in the time series data of the short-time power spectrum is norm normalized as an F / 2-dimensional vector. Here, F is the number of audio data (frame length) used when calculating a short-time power spectrum by FFT (Fast Fourier Transform). When each norm-normalized vector V (t) is a row vector, the autocorrelation matrix of the vector V (t) is expressed by equation (1).

このとき、スペクトルの全パワーの合計が所定の閾値未満しかない区間を除いた時刻の自己相関行列Ｐ（ｔ）を全て加算して相関行列Ｑを（２）式のように算出する。

At this time, the correlation matrix Q is calculated by adding all the autocorrelation matrices P (t) at times excluding a section where the total power of the spectrum is less than a predetermined threshold.

この相関行列Ｑを主成分分析して得られる固有ベクトルのうち、対応する固有値の大きい順に上位Ｍ位までの固有ベクトルを抽出すると、このＭ本の固有ベクトルで張られる部分空間（次元数Ｍの辞書部分空間）が、当該教示用音声データから生成される声紋辞書１２１となる。

Out of the eigenvectors obtained by performing principal component analysis on this correlation matrix Q, eigenvectors up to the M-th order are extracted in descending order of the corresponding eigenvalues. ) Is the voiceprint dictionary 121 generated from the teaching voice data.

  FIG. 16 is an explanatory diagram showing a method for generating a dictionary subspace and an input subspace in speaker identification processing. As shown in FIG. 16, the time series data 1402 of the short-time power spectrum uttered “Aiueo” is a column of the short-time power spectrum at each time. Among these, the short-time power spectrum excluding the section where the power is less than the predetermined threshold is the portions 1611 to 1615. The portions 1611 to 1615 correspond to “a”, “i”, “u”, “e”, “o” in order from the left. Using this as teaching data 1616, a dictionary subspace 1617 is generated.

  A subspace (an input subspace having a dimension number K) is also generated from the input audio data in the same manner. The time series 1401 of the short time power spectrum of the input voice is also a sequence of the short time power spectrum at each time as shown in FIG. Among these, the short-time power spectrum excluding the section where the power is less than the predetermined threshold is the portions 1621 to 1624. Using this as input data 1625, an input subspace 1627 is generated.

  FIG. 17 is an explanatory diagram showing the concept of the mutual subspace method in speaker identification processing. When collating with the voiceprint dictionary 121 in the speaker identification process, as shown in FIG. 17, the canonical angle 1731 of the input subspace 1627 and the dictionary subspace 1617 for each user is calculated as a similarity. The similarity is calculated by the “space method”.

  In step S1303 described above, the input subspace 1627 is collated with the teaching data 1616 of the dictionary subspace 1617 of all user IDs in the voiceprint dictionary 121 to calculate respective similarities, and the similarity is equal to or greater than a predetermined threshold. From the user ID with the highest similarity, the user IDs from the top N to the top N are extracted. In this way, by using a unit for collation with the voiceprint dictionary 121 as a partial space and learning and registering vowels in the voiceprint dictionary 121, speaker identification that is less influenced by the utterance content becomes possible.

  The mutual subspace method mentioned above is a non-patent document “Kazuhiro Fukui, Osamu Yamaguchi,“ Theoretical Extension of Subspace Method and Application to Object Recognition ”, IPSJ SIG, 2004-CVIM-145, pp219-228 , 2004 "is used.

(Face identification process)
Next, face identification processing by the face identification unit 105 will be described. FIG. 18 is a flowchart showing the procedure of face identification processing by the face identification unit 105.

  First, the face identifying unit 105 inputs an image frame captured by the omnidirectional camera 205 (step S1801), and collates the image frame with a face template indicating a region where a predetermined face image exists. A face area that is equivalent to or close to the face template is detected from the inside (step S1802). At this time, when there is a face area that has already been detected in the image frame input immediately before, face tracking processing for matching the face template only in the vicinity of the face area is first performed. Then, the face area search process is performed by collating the remaining area that is not the tracking target while scanning the face template.

  As a result, it is determined whether or not a face area has been detected (step S1803). If no face area has been detected (step S1803: No), input of the next image frame is awaited. On the other hand, when the face area is detected (step S1803: Yes), the coordinates of the image center position of the face area are obtained, and the coordinates of the image center position of the area are included with reference to the direction association table. An area number corresponding to “range” is set as direction information (step S1804).

  Next, the face image of the face area is compared with the face image pattern for each user ID registered in the face dictionary 122, and the degree of coincidence between the face image and the face image pattern of the user ID is determined for each user ID. The similarity shown is obtained, and a list of user IDs up to the top N ranks (N is an arbitrary integer) in the similarity is acquired (step S1805).

  Next, it is determined whether the maximum similarity in the list of user IDs is equal to or greater than a predetermined threshold (step S1806). If the maximum similarity is greater than or equal to the threshold (step S1806: Yes), the column consisting of the user IDs up to the top N and each similarity corresponding to the user ID is recognized as the face identification result, Face identification information composed of a column composed of each user ID and each similarity and the direction information set in step S1804 is generated and output (step S1807).

  On the other hand, if the maximum similarity is smaller than the predetermined threshold value in step S1806 (step S1806: No), the person unknown ID is set as the face identification result, and the face composed of the face identification result and the direction information set in step S1804. Identification information is generated and output (step S1808).

  Here, the collation between the face image and the face dictionary 122 performed in step S1805 will be described. After the face area is detected from the image frame for the first time, the size of the face image pattern that is successively cut out from each tracked face area is normalized to L × L pixels, and this is converted to G dimension (G = L × Norm normalization as a vector of L). Using a large number of norm normalization vectors obtained by tracking the same face image in time series, the correlation matrix is subjected to principal component analysis in the same manner as in the speaker identification process described above, and a subspace (input subspace) is obtained. )

  FIG. 19 is an explanatory diagram showing the concept of the mutual subspace method in face identification processing. As shown in FIG. 19, the face image pattern being tracked forms a face image pattern sequence 1941 obtained in time series, and an input subspace 1942 is generated using the face image pattern sequence 1941 as input data. Similarly, the face dictionary 122 generates a dictionary partial space 1944 using teaching face image pattern sequence 1943 obtained by actually capturing the face of the user as teaching data. In the face identification process, the similarity is calculated by a “mutual subspace method” in which the canonical angle 1945 of the input subspace 1942 and the dictionary subspace 1944 for each user (user ID) is calculated as the similarity.

  In step S1805 described above, the input subspace 1942 is collated with the face image pattern sequence 1943 of the dictionary subspace 1944 of all user IDs of the face dictionary 122 to calculate the respective similarities, and the similarity is a predetermined threshold value. From the user ID having the highest similarity, the user IDs from the top N to the top N are extracted.

(Region tracking process)
Next, the region tracking process by the region tracking unit 106 will be described. FIG. 20 is a flowchart showing the procedure of the area tracking process by the area tracking unit 106. When the region tracking unit 106 receives a region addition command from the identity verification unit 110 (step S2001), the region tracking unit 106 acquires an image template and an initial position from the received region addition command (step S2002). Here, the region addition command is a command for performing tracking including the image feature region to be tracked and the image template indicating the size, and the initial value of the image template, to the identity verification unit 110. The image feature area to be tracked is an image area that appears in the direction in which the face area is detected or an image area that appears in the direction in which the sound is detected, and is an area having a characteristic image.

  Upon receiving the region addition command, the region tracking unit 106 inputs an image frame after the time when the command is received (step S2003), and the region tracking unit 106 finds the most recent image template given in the vicinity of the initial position indicated by the region addition command. A sufficiently similar region is searched (step S2004).

  If such a region is detected as a result of the search (step S2005: Yes), the position of the detected new region in the detection direction is set as the initial position (step S2006), and the detection direction of the new region is set. Is output as area tracking information (step S2007). Thereafter, an image feature area is searched from the input image frame at this initial position.

  On the other hand, if an area most similar to the image template given in the vicinity of the initial position indicated by the area addition command is not detected in step S2005 (step S2005: No), the image feature of the image template is detected. The area is excluded from the tracking target (step S2008), and the fact that the tracking target is lost is output as area tracking information (step S2009).

  Here, the region tracking process will be described using a specific example. FIGS. 21A and 21B are schematic diagrams illustrating an example of setting the image feature area performed by the area addition command. FIG. 21A schematically shows the vicinity of the face detection direction and the vicinity of the voice detection direction in the whole sky image cut out from the wide-field whole sky image, and the face area is visible. In the example of FIG. 21A, a predetermined area 212 centering on the extracted face area 211 of the person 210 facing the front is set as an image feature area. Further, FIG. 21-2 shows a case where sound is heard, and the directional area 214 is set near the mouth of the person 213 who faces sideways and cannot see the face. A predetermined area 215 is set as the image feature area.

  22A and 22B are schematic diagrams illustrating an example of the area tracking process. FIG. 22-1 shows a state when the image template is generated. In this example, a predetermined area 222 centering on the extracted face area 221 of the person 220 facing the front is set as the image feature area. Next, as shown in FIG. 22-2, the face cannot be detected because the person 220 has started to move sideways, but this is achieved by using the image template generated in FIG. It is possible to track the movement of a person whose face has disappeared when the region 223 having the closest property is detected.

  Note that the above-described area tracking process is executed when the tracking unidirectional verification process by the tracking unidirectional verification unit 112 of the identity verification unit 110 is performed. When the same direction verification processing by the same direction verification unit 111 of the identity verification unit 110 is performed, the region tracking unit 106 does not track the image feature region, but acquires the previously obtained speaker identification information and face. The direction information set in the identification information is registered in the same direction verification unit 112 as the detection direction.

(Codirectionality verification process)
Next, the same direction verification processing by the same direction verification unit 111 of the identity verification unit 110 will be described. FIG. 23 is a flowchart showing the procedure of the same direction verification processing by the same direction verification unit 111.

  First, in the same direction verification unit 111, the area tracking unit 106 determines whether or not the expiration date of the already registered detection direction has passed (step S2301). Here, as the detection direction, the direction information set in the biometric identification information acquired earlier among the biometric identification information of the speaker identification information and the face identification information at the current time point is the region tracking unit 106 together with the expiration date. It is registered by. The expiration date indicates a period during which the biometric identification information can be used as the identity verification process.

  When the expiration date has passed (step S2301: Yes), the registered detection direction is deleted because it cannot be used for the identity verification process (step S2302), and new speaker identification information or new Is ready to acquire correct face identification information (step S2303).

  On the other hand, in step S2301, if the registered detection direction is within the valid period (step S2301: No), the detection direction is not erased and waiting for acquisition of new speaker identification information or new face identification information. A state is entered (step S2303).

  When new speaker identification information or new face identification information is acquired (step S2303: Yes), the direction information set in the acquired new speaker identification information or new face identification information is registered. Are detected (step S2304). Then, it is determined whether or not the difference in direction is equal to or less than a predetermined threshold (step S2305).

  When the difference in direction is equal to or smaller than a predetermined threshold (step S2305: Yes), the direction information of the newly acquired biometric identification information is the registered detection direction (direction information of the biometric information acquired earlier). Speaker identification information or face identification newly acquired by corresponding biometric identification information in the same set of information consisting of speaker identification information and face identification information detected in substantially the same direction The same set information is updated by being replaced with information and the validity period is extended (step S2306), and the same set information is output (step S2307).

  On the other hand, when the direction is larger than the predetermined threshold value in step S2305 (step S2305: No), it is determined that there is no detection direction that should be regarded as substantially the same, and the newly acquired speaker identification information or face The direction information set in the identification information is registered as a new detection direction (step S2308), and an area addition command designating this detection direction is output to the area tracking unit 106 (step S2309). Then, the same set information is updated with the newly acquired speaker identification information or face identification information and its expiration date (step S2310), and the same set information is output (step S2307).

  The newly registered detection direction in step S2308 is the newly acquired speaker identification information and face identification information until the expiration date is expired and the registration expires in steps S2301 and S2302 and the registration is deleted. Is used to determine direction identity.

  When the identity verification process is performed first, the same set information is obtained from the previously acquired biometric identification information and the newly acquired biometric identification information in the same set information update process (steps S2306 and S2310). Will be generated. In addition, when the vocabulary identification information in which the same direction information as the direction information set in the speaker identification information is output from the vocabulary identification unit 103, the vocabulary identification information is added to the same set information. . Further, when both the speaker identification information and the face identification information cannot be acquired, the same set information with the blank of the biometric identification information that could not be acquired is output. The same set information updated / generated in this way is used in dictionary update processing and user authentication processing described later.

  A specific example is given and demonstrated in the above-mentioned same direction verification process. As the direction of the input voice, an area number corresponding to the module number of the sound directivity forming module 102 is added to the direction information of the speaker identification information, and the direction of the input face image is also the face image in the image. An area number corresponding to the center position of the face is added to the face identification information. Therefore, in the autonomous mobile robot according to the present embodiment, the user is specified independently from each of the user's voice and the face image, and the direction of the user's voice and the direction of the user's face are independently set. Have acquired. In particular, the problem of identity verification due to simultaneity by the conventional user identification device is solved by specifying the direction of the voice and the face image independently.

  The autonomous mobile robot according to the present embodiment can identify the user from the voice and can determine from which direction the user is present only from the input voice. In addition, the autonomous mobile robot according to the present embodiment identifies a user from the captured face image, and grasps in which direction the user is present from only the captured face image. be able to. That is, the user is specified separately from the voice and the face image, and the direction in which the user exists is specified separately.

  Assuming that the user (or unknown person) identified in this way hardly moves, even if the user's face and voice are detected at intervals, they are detected in almost the same direction. By doing so, it can be considered that both belong to the same person. In addition, if a voice is heard during a period in which the user's face image can be detected or a face image is also detected during a period in which the user can hear the voice, the two are naturally detected in substantially the same direction. It can be considered that of a person, and it is possible to verify identity by the same direction.

  The verification of identity based on the same directionality means that even if the user identification result based on the face image and the user identification result based on the voice do not match, both are the same in the direction, so both are biometric information of the same person. It becomes an effective means for knowing that the problem is that the two identification results do not match, and can be used to update the biological dictionary.

  For example, there is a user A facing the autonomous mobile robot according to the present embodiment, and the autonomous mobile robot detects a face image of the user A, and the autonomous mobile robot Consider a case where a user A who is in the middle of business talks to the robot and says, "Do you receive an email?" In this case, the user's voice comes from the direction in which the face is detected. FIG. 24A is a schematic diagram illustrating a case where the user's voice comes from the direction in which the face is detected. As shown in FIG. 24A, the face of the user A is in the direction of a point 2441 on a virtual dome 2490 set around the main body 201 of the autonomously moving bot, and has a certain directivity range. Therefore, the voice of the user A is detected in the direction of the center point 2442. At this time, the same direction verification unit 111 of the identity verification unit 110 includes the point 2442 in the circular vicinity region 2443 having a predetermined radius set around the point 2441, that is, the face identification information acquired earlier. If the direction information of the face image and the direction information of the speaker identification information acquired later match, and the face image and the sound detection direction are determined to be substantially the same, the user of the face image and the sound Are verified to be the same person.

  On the other hand, for example, when there is a user A facing the autonomous mobile robot according to the present embodiment and the autonomous mobile robot detects a face image of the user A, the user A Let's consider a case where another user B who is facing sideways and cannot see his face utters "You have received an email?" FIG. 24-2 is a schematic diagram illustrating a case where another user's voice comes from a direction different from the direction in which the face is detected.

  In this case, when the user A and the user B are located azimuthally apart, the voice comes from a direction different from the direction in which the face image is detected. That is, as shown in FIG. 24-2, there is a face of the user A in the direction of the point 2444 on the virtual dome 2490 set around the main body 201 of the autonomous mobile bot, and the user A is away from the point 2444. The voice of the user B is detected from the direction of the center point 2445 of the directivity range. In a conventional user identification device that performs identity verification by conventional simultaneity, even in such a case, the voice detected in a short period of time when the face image is detected. It is mistakenly judged that there is.

  However, in the autonomous mobile robot according to the present embodiment, since the identity is verified by the same directionality, the circular vicinity region having a predetermined radius set around the point 2444 by the directionality verification unit 111. It is determined that the point 2445 is not included in 2446, that is, the direction information of the face identification information and the direction information of the speaker identification information do not match, so that the detection direction of the face image and the sound is sufficiently separated from each other. From the same person.

  It should be noted that the radius of the circular vicinity region is set to a predetermined fixed value, for example, an angle of 5 degrees, but when a face image is detected, a predetermined calculation is performed based on the size of the face image. It can be configured to determine the radius of the circular neighborhood. For example, the radius of the circular vicinity region may be determined to be larger than the distance between the center of the face (near the head of the nose) and the center of the mouth (about 5 cm) and less than the size of one directivity range. Good.

  For example, consider a case where there is a user A facing the autonomous mobile robot of the present embodiment, and the autonomous mobile robot detects the face of the user A. Over time, the user A attracted attention and turned sideways, so the autonomous mobile robot could not detect the face of the user A, but after a while user A turned sideways. Let's consider the case of saying "This room is a little hot". In this case, since the voice comes from the direction in which the face was previously detected and identified, the direction information of the face identification information acquired earlier and the direction information of the speaker identification information acquired later are It is verified that the user of the face image and the user of the voice are the same person by determining that they coincide with each other and determining that the detection direction of the face image and the sound is substantially the same.

  As described above, according to the identity verification process based on the same direction by the same direction verification unit 111 according to the present embodiment, the voice and the face detection time are the same even if the detection time is not almost the same unless the user moves. Since verification of a person is possible, verification of the same person can be performed more accurately than a conventional user identification device that performs verification of identity by simultaneity.

(Tracking unidirectional verification process)
Next, the tracking unidirectional verification processing by the tracking unidirectional verification unit 112 of the identity verification unit 110 will be described.

  In the case where the user hardly moves, the identity of the user can be accurately determined by the above-described directionality verification processing by the directionality verification unit 111. However, since the user actually moves, it is impossible to verify the same person only with the same directionality. That is, when certain biometric information detected once is lost in the middle and the user moves in the meantime, another type of biometric information of the same person is detected from another direction by this movement, the same directionality In the verification process, the identity of both cannot be verified.

  For this reason, in the autonomous mobile robot according to the present embodiment, the tracking same direction in which identity verification is performed by determining whether or not the direction information of a plurality of pieces of biometric identification information matches while tracking the image feature region. A sex verification process is performed.

  The face can be detected only when it is facing the front, and the voice can be detected only when it is spoken. As described above, the biological information for the autonomous mobile robot to identify the user is not always detected even when the user is on the robot side.

  Therefore, in this embodiment, the movement of the user is tracked using another clue that can be detected more stably than the biological information. This clue for movement needs to be actually related to the voice and face as biological information, and in this embodiment, it is used based on the image characteristics observed in the vicinity of the voice and face detection direction. The movement of the person is tracked. Such image features correspond to voice and facial user's clothing, skin and hair, etc., and image features are more stable for a long time regardless of whether the user is facing sideways or holding his mouth. Can be detected. That is, using the detection of biological information as a trigger, the movement of the user is tracked by recognizing the image characteristic in the detection direction.

  In this way, since the movement of the user who is the sender of the biometric information is tracked based on the image feature that can be detected more stably than the biometric information, it is detected in a certain direction at a certain time. It is possible to grasp in which direction the biological information transmission source has moved at another time.

  FIG. 25 is a flowchart showing the procedure of the tracking unidirectional verification process by the tracking unidirectional verification unit 112.

  First, the tracking direction verification unit 112 acquires tracking information from the area tracking unit 106 (step S2501). Then, it is determined whether or not the acquired tracking information includes the detection direction of the image feature area (step S2502). Here, the image feature area in the tracking information is the image feature area generated in step S2510 described later from the biometric identification information acquired previously.

  If it is determined that the detection direction of the image feature region is included in the tracking information (step S2502: Yes), the initial position of the image feature region being tracked is updated with the detection direction of the acquired tracking information. (Step S2503), the system waits for acquisition of new speaker identification information or new face identification information (Step S2505).

  On the other hand, if it is determined in step S2502 that the detection direction of the image feature area is not included in the tracking information (step S2502: No), that is, the fact that the image feature area to be tracked is lost is included. The image feature area currently being tracked is deleted from the tracking target (step S2504), and a new speaker identification information or new face identification information is awaiting acquisition (step S2505).

  Then, when new speaker identification information or new face identification information is acquired (step S2505: Yes), the initial position of the image feature area being tracked and the acquired new speaker identification information or new face identification The direction information set in the information is collated (step S2506), and it is determined whether or not the difference in direction is equal to or less than a predetermined threshold (step S2507). Here, the new speaker identification information or the new face identification information becomes the biometric information acquired later with respect to the biometric identification information acquired previously from which the image feature region was generated.

  When the difference in direction is equal to or smaller than a predetermined threshold (step S2507: Yes), it is assumed that the direction information of the newly obtained biometric identification information is substantially the same as the detection direction of the image feature region, and the abbreviation. The same set information is obtained by replacing the corresponding biometric identification information in the same set information consisting of the speaker identification information and the face identification information detected in the same direction with the newly acquired speaker identification information or face identification information. Is updated (step S2508), and the same set information is output (step S2509).

  On the other hand, if the direction difference is larger than the predetermined threshold value in step S2507 (step S2507: No), an area that is a characteristic part of the image is extracted from the newly acquired speaker identification information or face identification information. A new image feature area is generated (step S2510), and an area addition command including an image template including the new image feature area and its size and an initial position of the newly generated image feature area is received by the area tracking unit 106. (Step S2511). As the initial position, the image center position coordinates of the image feature area are set. Note that the biometric identification information that is the generation source of the newly generated image feature area is the biometric identification information that is acquired first. Then, the same set information is updated with the newly acquired speaker identification information or face identification information (step S2512), and the same set information is output (step S2509).

  When the identity verification process is performed first, the same set information is obtained by the previously acquired biometric identification information and the newly acquired biometric identification information in the same set information update process (steps S2508 and S2512). Will be generated. In addition, when the vocabulary identification information in which the same direction information as the direction information set in the speaker identification information is output from the vocabulary identification unit 103, the vocabulary identification information is added to the same set information. . Further, when both the speaker identification information and the face identification information cannot be acquired, the same set information with the blank of the biometric identification information that could not be acquired is output. The same set information updated / generated in this way is used in dictionary update processing and user authentication processing described later.

(Dictionary update process)
Next, biometric dictionary update processing by the dictionary update unit 108 will be described. FIG. 26 is a flowchart illustrating a procedure of biometric dictionary update processing by the dictionary update unit 108. First, the dictionary update unit 108 inputs the same set information output from the identity verification unit 110 (step S2601), and checks whether speaker identification information exists in the same set information (step S2602). If the speaker identification information does not exist in the same set information (step S2602: No), the next set of information for the same set information is awaited. On the other hand, when the speaker identification information exists in the same set information (step S2602: Yes), it is further checked whether the face identification information exists in the same set information (step S2603).

  If the face identification information does not exist in the same set information (step S2603: No), the process waits for input processing of the next same set information. On the other hand, if the face identification information exists in the same group information (step S2603: Yes), it is checked whether or not the unknown person ID is set in the speaker identification result of the speaker identification information included in the same group information. (Step S2604).

  If the unknown person ID is set in the speaker identification result of the speaker identification information (step S2604: Yes), the unknown person ID is set in the face identification result of the face identification information included in the same set information. It is checked whether or not (step S2605).

  If the person unknown ID is set in the face identification result of the face identification information (step S2605: Yes), the user cannot be specified by the face identification information of the same set information or by the speaker identification information. The next input processing waiting state for the same set information is entered. On the other hand, when the person unknown ID is not set in the face identification result of the face identification information (step S2605: No), the voiceprint dictionary 121 is updated and strengthened (step S2606). Specifically, it is determined that the user identification is successful only by the face identification information, that is, the face, and in the voiceprint dictionary 121, the same set information is added to the voiceprint pattern corresponding to the user ID having the maximum similarity of the face identification information. The voiceprint dictionary 121 is updated and strengthened by registering the voice data included in.

  In step S2604, when the unknown person ID is not set in the speaker identification information of the same set information (step S2604: No), it is further checked whether the unknown person ID is set in the face identification information of the same set information. (Step S2607).

  If a person unknown ID is set in the face identification information of the same set information (step S2607: Yes), the face dictionary 122 is updated and strengthened (step S2608). Specifically, in this case, it is determined that the user identification is successful only by the speaker identification information, that is, only by the voice, and corresponds to the user ID having the maximum similarity of the face identification information in the face dictionary 122. The face dictionary 122 is updated and strengthened by registering image data included in the same set information for the face image pattern.

  If the unknown person ID is not set in the face identification information of the same set information in step S2607 (step S2607: No), it is determined that the user identification is successful by both the face and the voice, and the same set information It is checked whether or not the user ID having the maximum similarity set in the speaker identification information and the user ID having the maximum similarity set in the face identification information do not match (step S2609).

  If the user ID having the maximum similarity set in the speaker identification information of the same set information and the user ID having the maximum similarity set in the face identification information do not match (step S2609: Yes), the biological dictionary with a small maximum similarity is updated and strengthened (step S2610). Specifically, in this case, it is determined that the biological dictionary having the smaller maximum similarity is deteriorated, and the same set information is stored in the other biological dictionary corresponding to the user ID having the larger maximum similarity. Among the audio data and face image data included in the data, the data corresponding to this biometric dictionary is registered and the biometric dictionary is updated and strengthened.

  If the user ID having the maximum similarity set in the speaker identification information of the same set information matches the user ID having the maximum similarity set in the face identification information in step S2609 (step S2609). : No), it is determined that none of the biological dictionaries (voiceprint dictionary 121, face dictionary 122) is in a good state in which a sufficient degree of similarity is output without deterioration, Become.

  Here, the update of each biological dictionary is performed by the following method. As described above, the voiceprint dictionary 121 and the face dictionary 122 which are biological dictionaries are partial spaces generated from teaching data. At this time, the update of the biological dictionary adds data to be newly learned to the dictionary partial space that has already been generated. Therefore, the correlation matrix used when generating the dictionary partial space is used again. The new teaching data is subjected to the same processing as that for generating the dictionary subspace and is converted into an autocorrelation matrix. A new correlation matrix is generated by adding this autocorrelation matrix to the already registered correlation matrix, and eigenvectors up to the M-th order are extracted from the eigenvectors obtained by principal component analysis in descending order of the corresponding eigenvalues. . As a result, the subspace spanned by the newly extracted M eigenvectors becomes a dictionary subspace reflecting new teaching data in addition to the teaching data so far.

  In this way, identification of each piece of biological information is achieved by including speaker identification information, voice data, face identification information, and image data relating to a plurality of types of biological information (speech and facial images) relating to the same user as the same set of information. By comparing the results and detecting a deteriorated biometric dictionary related to the same user, and using the biometric information of the user being input as teaching data, the deteriorated biometric dictionary is automatically re-established. Can be strengthened.

(User authentication processing)
Next, user authentication processing by the user authentication unit 107 will be described. FIG. 27 is a flowchart illustrating a procedure of user authentication processing by the user authentication unit 107. First, the user authentication unit 107 inputs the same set information output from the identity verification unit 110 (step S2701), and sets the unknown person ID in the speaker identification result of the speaker identification information included in the same set information. It is checked whether or not it is done (step S2702).

  If the unknown person ID is set in the speaker identification result of the speaker identification information, it is further checked whether or not the unknown person ID is set in the face identification result of the face identification information (step S2703). If the person unknown ID is set in the face identification result of the face identification information (step S2703: Yes), the user cannot be specified by voice or by face, so Wait for input.

  On the other hand, in step S2703, when the person unknown ID is not set in the face identification result of the face identification information (step S2703: No), it is determined that the user identification is successful only by the face, and the same group information The user ID having the maximum similarity set in the face identification information is set as the authentication result (step S2704), the user ID is set in the user authentication information, and the user authentication information is output (step S2708).

  In step S2702, if the unknown person ID is not set in the speaker identification result of the speaker identification information included in the same set information (step S2702: No), the unknown person ID is further included in the face identification result of the face identification information. Is checked (step S2705).

  If a person unknown ID is set in the face identification result of the face identification information (step S2705: Yes), it is determined that the user identification is successful only by voice, and the speaker identification information of the same set information Is set as the authentication result (step S2706), the user ID is set in the user authentication information, and the user authentication information is output (step S2708).

  On the other hand, if the unknown person ID is not set in the face identification result of the face identification information in step S2705 (step S2705: No), it is determined that the user identification is successful by both the face and the voice, A user ID having the largest similarity between the maximum similarity set in the speaker identification information of the same set information and the maximum similarity set in the face identification information is set as an authentication result (step S2707). This user ID is set in the user authentication information and the user authentication information is output (step S2708).

  The user authentication information is input by the service providing unit 109, and various service processes such as an email check service are executed using the user ID set in the user authentication information by the service providing unit 109.

  As described above, in the autonomous mobile robot according to the present embodiment, the direction in which the user exists is obtained separately from the audio data obtained from the microphones 206, 207, and 208 and the image data obtained from the all-sky camera 205. Then, among the speaker identification information that is the biometric information for the voice data and the face identification information that is the biometric information for the image data, an image that is a characteristic image region with respect to the image in the direction indicated by the previously obtained biometric identification information The region that approximates the feature region is tracked, and it is determined whether the difference between the detection direction of the image feature region and the direction of the biometric identification information acquired later is equal to or less than a predetermined threshold value. In the following cases, it is determined that the user identified by the biometric identification information acquired earlier is the same as the user identified by the biometric identification information acquired later, so multiple types of biometric information are used. When the user is identified, the identity of the user can be verified based on the identity of the direction of the user obtained from each separate biometric information, and a different biometric from the user after a certain period of time Take information Even when it is possible to verify the identity of users with high accuracy.

  Further, in the autonomous mobile robot according to the present embodiment, the same set information is generated by pairing the speaker identification information and the face identification information whose identity is verified with high accuracy in this way. Since the biological dictionary is updated according to the contents of the group information, the necessity of updating the biological dictionary can be determined more accurately.

  Further, in the autonomous mobile robot according to the present embodiment, the user is authenticated by authenticating the user from the same set of information including the speaker identification information and the face identification information whose identity is verified with high accuracy. Since the service for the user is provided, the service can be provided to the user more safely.

  In this embodiment, an example in which the user identification device of the present invention is applied to an autonomous mobile robot is shown. However, the present invention is not limited to this, and the user identification function can be applied to other than the autonomous mobile robot. The user identification device according to the present invention can be applied to any device having the above.

It is a block diagram which shows the functional structure of the autonomous mobile robot concerning this Embodiment. It is a schematic diagram which shows the external appearance of the autonomous mobile robot of this Embodiment. It is a schematic diagram which shows arrangement | positioning of the all-sky camera 205 which images two users. It is a schematic diagram which shows the all-sky image which imaged two users of arrangement | positioning shown in FIG. 2 is a block diagram illustrating a configuration of an acoustic directivity forming unit 101. FIG. 3 is a block diagram illustrating a configuration of an acoustic directivity forming module 102. FIG. It is explanatory drawing which showed the example of the directivity center of the sound directivity formation module. It is a schematic diagram which shows the directivity range set by the front beamformer and the rear beamformer of the acoustic directivity module from above the robot. It is a schematic diagram which shows the directivity range set by the beam former of a front | former stage of a sound directivity module, and the beam former of a back | latter stage from a horizontal direction. It is explanatory drawing which showed the example of arrangement | positioning of the directivity range which covers 180 degree | times by 20 degree | times. It is explanatory drawing which shows the example before correction | amendment in the correction | amendment from a relative azimuth | direction to an absolute azimuth | direction in an autonomous mobile robot. It is explanatory drawing which shows the example after correction | amendment in the correction | amendment from a relative azimuth | direction to an absolute azimuth | direction in an autonomous mobile robot. 3 is a data structure diagram of directional sound information output from an acoustic directivity forming module 102 of the acoustic directivity forming unit 101. FIG. It is a data structure figure of vocabulary identification information. It is a data structure figure of speaker identification information. It is a data structure figure of face identification information. It is a data structure figure of a direction matching table. It is a data structure figure of the same set information. It is a flowchart which shows the procedure of the whole process of the user identification by the autonomous mobile robot concerning this Embodiment. 5 is a flowchart illustrating a procedure of directivity audio information generation processing by an acoustic directivity forming unit 101. 5 is a flowchart illustrating a procedure of vocabulary identification processing by a vocabulary identification unit 103. 5 is a flowchart illustrating a procedure of speaker identification processing by a speaker identification unit 104. It is explanatory drawing which shows an example of the audio | voice data converted into the power spectrum row | line | column from the input audio | voice data. It is explanatory drawing which shows an example of the voiceprint pattern 1402 when the man A utters "Aiueo". It is explanatory drawing which shows an example of the voiceprint pattern 1403 when the woman B utters "Aiueo". It is explanatory drawing shown about the production | generation method of the dictionary subspace and input subspace in a speaker identification process. It is explanatory drawing which showed the concept of the mutual subspace method in a speaker identification process. 5 is a flowchart illustrating a procedure of face identification processing by a face identification unit 105. It is explanatory drawing which shows the concept of the mutual subspace method in face identification processing. 5 is a flowchart showing a procedure of region tracking processing by the region tracking unit 106; It is a schematic diagram which shows the example of a setting of the image characteristic area performed by an area | region addition command. It is a schematic diagram which shows the other example of an image feature area | region performed by an area | region addition command. It is a schematic diagram which shows the example of an area | region tracking process. It is a schematic diagram which shows the other example of an area | region tracking process. It is a flowchart which shows the procedure of the same direction verification process by the same direction verification part 111. FIG. It is a schematic diagram which shows the case where a user's audio | voice has arrived from the direction from which the face is detected. It is a schematic diagram which shows the case where another user's audio | voice comes from the direction different from the direction from which the face is detected. It is a flowchart which shows the procedure of the tracking directionality verification process by the tracking directionality verification part 112. It is a flowchart which shows the procedure of the update process of the biometric dictionary by the dictionary update part. It is a flowchart which shows the procedure of the user authentication process by the user authentication part 107.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Acoustic directivity formation part 102 Acoustic directivity formation module 103 Vocabulary identification part 104 Speaker identification part 105 Face identification part 106 Area tracking part 107 User authentication part 108 Dictionary update part 109 Service provision part 110 Identity verification part 111 Same direction Sex verification unit 112 Tracking unidirectional verification unit 120 Biological dictionary storage unit 121 Voice print dictionary 122 Face dictionary 123 Vocabulary dictionary 201 Body 202, 203 Driving wheel 205 All-sky camera 206, 207, 208 Microphone

Claims (11)

Detecting means for detecting a user's voice and outputting the detected voice as user's biological information;
Imaging means for capturing an image of the user and outputting the captured image as biological information of the user;
Biometric dictionary storage means for storing a plurality of biometric dictionaries in which user identification information and user biometric information are associated with each other, for each biometric information output by the imaging means and the detection means;
A user is identified from the biometric information based on the biometric information output by the detection means and the biometric dictionary corresponding to the biometric information, and is associated with the user identification information and the direction in which the user exists. Biometric identification information is generated, a user is identified from the biometric information based on the biometric information output by the imaging unit and the biometric dictionary corresponding to the biometric information, and the user identification information and the user exist Identification means for generating biometric identification information associated with the direction to perform,
An area that approximates an image feature area that is a characteristic area with respect to an image in the direction indicated by the first biometric identification information that is generated by the identification means and that is previously acquired is detected from an image that is newly input by the imaging means. Then, while setting the detected area as the new image feature area, area tracking means for obtaining the detection direction of the set new image feature area,
Whether or not the difference between the detection direction of the image feature area obtained by the area tracking unit and the direction of the second biometric identification information acquired after the first biometric identification information is equal to or less than a predetermined threshold value. When the difference in direction is equal to or smaller than the threshold, it is determined that the user identified by the first biometric identification information is the same as the user identified by the second biometric identification information. Identity verification means;
A user identification device comprising:   In the identity verification means, the difference between the direction of the first biometric identification information and the direction of the second biometric identification information acquired after the first biometric identification information is not more than a predetermined threshold value. The user identified by the first biometric identification information is the same as the user identified by the second biometric identification information when the difference in direction is equal to or less than the threshold value. The user identification device according to claim 1, wherein the user identification device is determined.   The identity verification means further includes the first biometric identification determined that the user identified by the first biometric identification information and the user identified by the second biometric identification information are the same. The user identification device according to claim 1, wherein the same set information in which the information is associated with the second biological identification information is generated.   The identification unit further obtains, for each user, a similarity indicating a degree of coincidence between the biometric information for each user registered in the biometric dictionary and the biometric information output by the detection unit. 4. The user identification apparatus according to claim 3, wherein the identification information of the user having a degree, the maximum similarity, and the biometric identification information associated with the direction in which the user exists are generated. It is determined whether or not the user having the maximum similarity indicated by the first biometric identification information associated with the same set information matches the user having the maximum similarity indicated by the second biometric identification information. And a dictionary updating means for updating the biometric dictionary corresponding to the biometric identification information having a small maximum similarity when the values do not match based on the biometric identification information having a large maximum similarity,
The user identification device according to claim 4, further comprising: The user having the maximum similarity of the biometric identification information having the greatest similarity among the first biometric identification information and the second biometric identification information associated with the same set information is associated with the same set information. A user authentication means for authenticating the user of the attached biometric information;
Service providing means for executing predetermined service processing for the user authenticated by the user authentication means;
The user identification device according to claim 4, further comprising: The detection means includes first, second and third voice input means for detecting voice of a user from different directions and outputting voice information of the detected voice as the biological information.
From the first voice information output from the first voice input means and the second voice information output from the second voice input means, the voice is input to the first directivity range in a predetermined direction range. The fourth voice information with limited sensitivity is output, and the first directivity range is determined from the output fourth voice information and the third voice information output from the third voice input means. Further, acoustic directivity forming means for outputting fourth voice information whose input sensitivity is limited to the limited second directivity range, and estimating the second directivity range as an arrival direction of the user's voice;
The user identification device according to claim 1, further comprising: The acoustic directivity forming means further outputs directional voice information in which the arrival direction and the fourth voice information are associated with each other,
The biological dictionary storage means stores a speaker dictionary associating user identification information with the user's voice information as the biological dictionary,
The identifying means identifies the user based on the directional speech information output by the acoustic directivity forming means and the speaker dictionary, and the identification information of the user and the direction of arrival of the directional speech information, The user identification device according to claim 7, further comprising speaker identification means for generating speaker identification information in association with each other as the first biometric identification information. The biological dictionary storage means stores a face dictionary in which identification information of a user is associated with an image of the user's face as the biological dictionary,
The identification means searches the user's face area from the image based on the image output by the imaging means and the face dictionary, identifies the user, and identifies the user's identification information and the face area. The user identification device according to claim 7, further comprising face identification means for generating face identification information associated with the image center position as a direction as the second biometric identification information. Detecting the user's voice, and outputting the detected voice as the user's biometric information, the biometric information output by the detection means, the user identification information corresponding to the biometric information, and the user's biometric information A user is identified from the biometric information based on the associated biometric dictionary, biometric identification information associated with the user identification information and the direction in which the user exists is generated, and an image of the user is captured. Identifying the user from the biometric information based on the biometric information output by the imaging means that outputs the captured image as the biometric information of the user and the biometric dictionary corresponding to the biometric information, and identifying the user And an identification step for generating biometric identification information associated with the direction in which the user exists,
An area that approximates an image feature area that is a characteristic area with respect to the image in the direction indicated by the first biometric identification information that is generated in the identification step and that has been previously acquired is detected from the image that is newly input by the imaging unit. An area tracking step for setting the detected area as a new image feature area and obtaining a detection direction of the set new image feature area;
Whether or not the difference between the detection direction of the image feature region obtained by the region tracking step and the direction of the second biometric identification information acquired after the first biometric identification information is equal to or less than a predetermined threshold value. And the user identified by the first biometric identification information is the same as the user identified by the second biometric identification information when the direction difference is equal to or less than the threshold. An identity verification step;
A user identification method comprising: Detecting the user's voice, and outputting the detected voice as the user's biometric information, the biometric information output by the detection means, the user identification information corresponding to the biometric information, and the user's biometric information A user is identified from the biometric information based on the associated biometric dictionary, biometric identification information associated with the user identification information and the direction in which the user exists is generated, and an image of the user is captured. Identifying the user from the biometric information based on the biometric information output by the imaging means that outputs the captured image as the biometric information of the user and the biometric dictionary corresponding to the biometric information, and identifying the user And an identification step for generating biometric identification information associated with the direction in which the user exists,
An area that approximates an image feature area that is a characteristic area with respect to the image in the direction indicated by the first biometric identification information that is generated in the identification step and that has been previously acquired is detected from the image that is newly input by the imaging unit. An area tracking step for setting the detected area as a new image feature area and obtaining a detection direction of the set new image feature area;
Whether or not the difference between the detection direction of the image feature region obtained by the region tracking step and the direction of the second biometric identification information acquired after the first biometric identification information is equal to or less than a predetermined threshold value. And the user identified by the first biometric identification information is the same as the user identified by the second biometric identification information when the direction difference is equal to or less than the threshold. An identity verification step;
User identification program that causes a computer to execute.

Images