JP4770869B2 - Robot device, verification environment determination method, and verification environment determination program - Google Patents

Description

The present invention relates to a robot apparatus, a verification environment determination method, and a verification environment determination program that are used at home and have a function of storing and collating a user's face using image information.

A robot apparatus typified by a recent pet-type robot is sometimes used for entertainment purposes, and is configured to register an owner's face image, collate a person from the captured image, and call a name.

A configuration example of a conventional robot apparatus having a face matching function is shown in FIG. This is because the image pickup means 1 for picking up an image, the face area of a person is detected from the image obtained by the image pickup means 1 and stored, and the newly acquired image is stored using the stored face image information. Face detection collation means 2 for collating a person, information notification means 3 for notifying the user of the collation result obtained by the face detection collation means 2, and whether the collation result notified to the user by the information notification means 3 is correct or incorrect. The information input means 4 input by the user, the control means 7 for controlling the robot, the traveling means 8 for moving the robot itself, and the like. The robot interacts with the user using the information input unit 4 and the information notification unit 3 based on the command from the control unit 7, and moves within the room using the traveling unit 8. Then, the user's face is recognized by using the imaging unit 1 and the face detection / collation unit 2, and it is determined who is interacting with the user.

When face collation is performed with a robot used at home, the lighting environment at home becomes a problem. The interior of a house often has a single lighting in the room, or is designed to easily capture sunlight into the room, so the lighting fluctuations depending on the location and time zone are severe. In addition, since it is relatively dark compared to an office or the like that is designed for lighting in order to perform detailed work, there is an extremely poor verification environment. In the conventional collation method, when the feature amount of the face image is fluctuated due to the illumination variation caused by such illumination environment variation, the variation of the feature amount cannot be absorbed and the collation performance accuracy is lowered. In addition, when the face area is not sufficiently illuminated, the features necessary for matching cannot be extracted and matching may not be successful.

As a method for performing face matching under such a condition where the illumination environment varies greatly, there is a technique disclosed in Japanese Patent Laid-Open No. 2000-222576. The technology disclosed in this publication stores feature amounts extracted from a plurality of images with different lighting per person as dictionary data, and absorbs variations in illumination by performing linear discrimination using them. .

JP 2000-222576 A

However, even with the technique disclosed in Japanese Patent Laid-Open No. 2000-222576, the face image acquired in an illumination state that is significantly different from that at the time of registration cannot absorb the fluctuation and cannot be correctly verified. There is.

The present invention has been made in view of the above circumstances, and when face matching is performed in a place such as a general home where there is a large variation in lighting or where there is a lighting environment that is extremely inferior for matching, an error in face matching is detected. An object of the present invention is to provide a robot apparatus, a verification environment determination method, and a verification environment determination program that can be reduced.

In order to achieve such an object, the present invention provides a robot apparatus including face detection / collation means for collating a person using face image information with respect to an input image, the illumination environment state observation detecting a lighting environment state Means for determining whether the lighting environment state is suitable for collation, and the lighting environment state is suitable for collation with reference to a collation performance recording unit in which the judgment unit records the success or failure of the collation for the lighting environment state. If the lighting environment state is determined not to be suitable for collation, the face detection / collation unit does not perform collation.

As is apparent from the above description, the present invention can reduce collation errors by determining the lighting environment and not performing collation in an illumination environment that is not suitable for collation. Therefore, face matching can be performed with few errors in a home where there is an environment that is inferior for face matching, such as a large illumination fluctuation and extremely dark.
Also, when checking the lighting environment, check the lighting environment at what rate in the past in the lighting environment, and if there are few correct answers, move the robot without checking and move the position. Or change direction. There are various variations in lighting environments and face registration states in different homes, and it is difficult to design a robot by accurately predicting in advance in which lighting environment matching will be successful. Therefore, if the user actually learns where and in what lighting conditions the matching process does not work in the face matching process, the matching will be performed more accurately than if the lighting conditions are not good in advance. You can determine when you are good at or not.
In addition, the past matching performance is searched from the matching performance recording means, and the total of the number of times the face matching result is correct in the lighting category in which the time, curtain state and lighting device state are the same as the current state is checked. When the value divided by the cumulative number of times is equal to or greater than the value in the current lighting category, movement is performed to a position that is equal to or greater than the threshold value or highest. In this way, the user can be guided to a good lighting environment, and face matching errors can be reduced.
Also, the lighting state when the face registration is performed for each user is stored, and if the lighting condition is such that the user has not registered the face in the past while the user is interacting with the robot, the user's face Start registration or urge the user to register face. By doing so, it is possible to register face images under various illumination conditions more efficiently.
In addition, face images are registered for each of various illumination states, and when collation is performed, the illumination state at that time is observed, and collation is performed using a dictionary created from images registered in the same illumination state. By doing in this way, collation can be performed with higher accuracy than when registration / collation is performed without evaluating the lighting environment.

Next, embodiments of the present invention will be described in detail with reference to the accompanying drawings. 1 to 19 show an embodiment according to the present invention.
(First embodiment)
(Description of configuration)
Referring to FIG. 1, a first embodiment according to the present invention is a CCD camera that captures an image and digitizes it, or an image pickup means 1 comprising an analog image pickup device and an AD converter, and an image obtained by the image pickup means 1. When the control unit 7 determines the transition to the state in which face matching is performed, the person matching of the newly acquired image is performed using the previously stored face image information. The face detection collating means 2 to be performed, the information notifying means 3 for notifying the user by speaking the voice of the collation result obtained by the face detection collating means 2 or displaying it on the CRT, and the information notifying means 3 Information input means 4 for the user to input whether the verification result notified to the user is correct or incorrect by voice, button, switch, or the like, and verification result reliability calculation for determining whether the illumination state is suitable for face verification Means 5, robot state observation means 6 for holding information such as the position and time in the robot room, control means 7 for controlling the robot, and traveling means such as wheels and motors for the robot itself to move 8 and.

The face detection / collation unit 2 detects a person's face from the image captured by the imaging unit 1 and registers it, or when the control unit 7 determines a transition to a state in which face collation is performed, the face detection / collation unit 2 The person verification is performed using a face image stored in advance. Here, as an example of a method for detecting a human face image from an image, a method disclosed in Japanese Patent Application Laid-Open No. 2000-222576 will be described. First, a pixel group that moves in the screen is detected. A difference image g is generated by taking the difference between the input image data and the image data input immediately before. Furthermore, the difference image g of past m frames (m is an integer of 2 or more) is added, and an average difference is obtained to obtain an integrated difference image G. The integrated difference image G has a pixel value of 0 in a non-motion area, and a larger pixel value in a movement area. Since the integrated difference image G contains a large amount of sesame salt noise, noise removal processing is performed. Examples of noise removal processing include expansion / contraction processing and median filter processing. Next, the head rectangular coordinates are obtained from the integrated difference image G. The motion area width is obtained for each scan line. The motion region width represents the difference between the maximum and minimum x-coordinate values of the motion region in each scan line. Next, the y coordinate of the head is obtained. As a method of obtaining the vertex coordinates, there is a method in which the minimum value of the y coordinate of the motion region is used as the vertex. Next, the y coordinate of the base of the head rectangle is obtained. As a method of obtaining the base coordinates of the head rectangle, a search is performed from the top to the bottom (y direction), a line having a motion area width smaller than the average value dm of the motion area width is obtained, and the Y coordinate is within that line. You may use the method which makes the largest place the base of a head rectangle. Next, the left and right x coordinates of the head rectangle are obtained. As a method for obtaining the left and right x-coordinates, a method is used in which the coordinates of the left and right ends of the motion region in the line having the largest motion region width in the range from the top of the head to the lower part of the head are used. The head rectangle obtained in this manner is cut out with a rectangle of an appropriate size while shifting one pixel at a time, and the likelihood of a face is evaluated, and the most likely face position is detected as a face. For evaluation of facial appearance, first, a face image is accurately cut out manually from a large number of images of various persons prepared in advance, and this cut out face image data is subjected to principal component analysis to be stretched with low-order eigenvectors. Get subspace. Then, a determination is made based on whether or not the distance between the partial space spanned by the low-order eigenvector and the rectangular image cut out at the appropriate size described above is short. In addition, the document "Kosugi Shin," Finding and locating faces in scenes using multiple pyramids for personal identification ", IEICE Transactions, Vol. J77-D-II, No4, pp.672 -681, April 1994 ", it is also possible to apply the method using template matching by the light and shade pattern.

The face image matching method is obtained as a result of principal component analysis of a large number of face images as described in the literature “M. Turk, A. Pentland,“ Face Recognition on Using Eigenfaces ”, Proceedings of IEEE, CVPR91”. The feature vector is the component of the base vector that projects the input face image into the subspace spanned by the lower-order ones. The feature vector of the image to be collated and the features of the registered image representing each registered person A method of searching for the Euclidean distance from the vector and using the person with the smallest Euclidean distance as the identification result may be applied. Alternatively, a pattern identification method may be used by obtaining the degree of similarity with the target face image using a linear discrimination dictionary created from human feature data disclosed in Japanese Patent Laid-Open No. 2000-222576.

The information notification means 3 is controlled by the control means 7 and notifies the user of the collation result when the face collation result is obtained by the face detection collation means 2 and asks for confirmation of whether the collation result is correct. For example, the user's name as a result of collation is spoken from a speaker using voices recorded in advance or synthesized voice by a computer, or the user's name or mark is presented on a CRT or liquid crystal monitor.

The information input unit 4 is controlled by the control unit 7 and acquires a user's answer to the collation result notified by the information notification unit 3. For example, it is possible to identify whether the user is “yes” or “no” by voice recognition, or to allow the user to input whether the answer is correct by pressing a button.

The collation result reliability calculation unit 5 determines whether or not the illumination environment when performing face collation is suitable for collation from the information from the imaging unit 1 and the robot state observation unit 6.

The following can be used as the quantities characterizing the lighting environment.
[Information from the imaging means 1]
1. Contrast of the face area (dispersion of luminance values of the face area obtained by the face detection collating means 2)
Depending on the lighting environment, the contrast of the detected face area may be large or small. In general, when the face is backlit or the face is hardly illuminated, the contrast of the face area becomes small. If the contrast of the face area is small, facial features will be lost, making it impossible to collate with high accuracy. The contrast of the face area changes depending on the environment, and has a great influence on the matching performance.
2. Ratio of the average brightness of the face area and the background (ratio of the average brightness value of the face area obtained by the face detection collating means 2 and the average brightness value of the outer neighboring area)
The ratio between the average luminance value of the face area and the background increases when the backlight is on. When the backlight is used, the contrast is reduced and the detection accuracy is lowered.
3. Average brightness value of the entire image The average brightness value of the entire image roughly represents the brightness of the entire room. The brightness of the room is related to the amount of light hitting the face.

According to the determination method of these conditions, it can be determined that the contrast of the face area 1 is more suitable for collation as the feature amount is larger. The ratio of the average luminance value of the face area 2 to the background and the image 3 It can be determined whether or not the average of all luminance values is suitable for collation based on whether or not the upper limit and the lower limit are within a certain range.

[Information obtained from other than the camera]
1. Time of day (morning / daytime / evening / night)
The lighting environment is greatly different between the case where there is natural light entering from the outdoors in the daytime and the case where it is illuminated only by the lighting of the room at night.
2. Whether or not the lighting device is lit (the lighting device state observation means (not shown) is linked to the lighting device installed in the room, and the robot (robot state observation means 6) acquires whether or not the lighting device is lit)
3. Whether the curtain is open (the curtain state observation means (not shown) is linked to the curtain installed in the room, and the robot (robot state observation means 6) acquires whether the curtain is open).
Information on whether or not the lighting device is lit and whether or not the curtain is open is greatly related to the presence or absence of illumination light and natural light, which are the main light sources in the home.
4). Position in the robot's room (obtains robot position using information from wheel encoders and images)
Since the position where illumination and natural light enter is fixed in the room, where the robot is in the room and in what direction the person is observed is largely related to the light on the face.

The determination method for these conditions is suitable for verification by determining the lighting environment at the current robot position at the time of verification from information on whether the lighting device is lit and whether the curtain is open. Determine whether or not.

The robot state observing means 6 acquires information from other than the camera that is used by the verification result reliability calculating means 5 to determine the illumination environment. For example, a wheel encoder for determining the position of the robot in the room, a room state observing means (not shown) for determining whether the light is lit in conjunction with a lighting device installed in the room, It consists of one or a plurality of curtain state observation means (not shown) for determining whether or not the curtain is open in conjunction with the curtain.

The control means 7 controls the robot. The traveling means 8 includes a motor, an axle, wheels, and the like, and realizes movement of the robot in the room.

(Description of operation)
Next, the operation procedure of this embodiment will be described with reference to the flowchart shown in FIG. In the first embodiment, first, when a transition to a state for identifying a face is determined by the control means 7 (step S201), an illumination environment when the face matching is to be performed by the matching result reliability calculation means 5 Is obtained from information from the imaging means 1 and the robot state observation means 6, and it is determined whether or not it is suitable for collation (step S202).

First, the case where it is determined in step S202 that the reliability is high (step S202 / YES) will be described. In this case, the face detection / collation unit 2 detects and collates the face image from the image obtained by the imaging unit 1, and associates it with someone registered in advance (step S203). Let the resulting person be L. Thereafter, the information notification means 3 informs the user of the collation result and asks the user to confirm whether the collation result is correct (step S204).

Next, the case where it is determined in step S202 that the reliability is low (step S202 / NO) will be described. In this case, the robot body is moved using the traveling means 8 without performing face matching (step S205), and the illumination environment is again observed and measured from the imaging means 1 and the robot state observation means 6. If the condition of step S202 is not satisfied even if the movement is performed many times, a realization method of an operation flow for performing face matching is also conceivable.

As described above, according to the present embodiment, it is possible to reduce verification errors by avoiding verification in an illumination environment that is determined based on knowledge of lighting conditions and is not suitable for verification. Specifically, if the detected image of the face area does not have sufficient contrast, or if the room lighting device is not attached, it is determined that the image is not suitable for collation and the robot is moved without performing collation. The occurrence of misjudgment can be reduced by changing the position and direction.

(Second Embodiment)
(Description of configuration)
A second embodiment according to the present invention will be described in detail with reference to the drawings. Referring to FIG. 3, the second embodiment of the present invention is based on a CCD camera that captures and digitizes an image, or an imaging unit 1 that includes an analog imager and an AD converter, and an image obtained by the imaging unit 1. When a face area of a person is detected and stored, and the transition to a state in which face matching is performed is determined by the control means 7, person matching of a newly acquired image is performed using face image information stored in advance. The face detection / collation unit 2, the information notification unit 3 for notifying the user by speaking the voice of the collation result obtained by the face detection / collation unit 2 or displaying it on the CRT, etc. Face matching is performed based on information from the information input means 4 in which the user inputs voice, buttons, switches, or the like, and information from the imaging means 1 and the robot state observation means 6 as to whether the notified verification result is correct or incorrect. Lighting environment determination means 500 for determining whether the lighting environment is suitable for collation, robot state observation means 6 for holding information such as the position and time in the robot room, and the robot is controlled. Control means 7 and a traveling means 8 such as a wheel or a motor for the robot itself to move.

The lighting environment determining unit 500 determines the lighting environment based on the image information obtained from the imaging unit 1, the time obtained from the robot state observing unit 6, and the information on the current position, and the lighting environment is a predetermined category. The number of successful face matching and the number of face matching are recorded for each category according to the lighting environment judgment means 501 for judging which one of them corresponds, and the matching result correct / incorrect information obtained by the information input means 4 Collation performance recording means 502 for performing the verification, and lighting environment comparison means 503 for determining whether the current lighting environment determined by the illumination environment determination means 501 is suitable for collation based on the information recorded in the collation performance recording means 502. And having.

The illumination environment determination unit 501 determines the illumination environment when performing face matching from information from the imaging unit 1 and the robot state observation unit 6. Since the determination material for determining the illumination environment is the same as that in the first embodiment described above, the description thereof is omitted. The lighting environment is expressed using one or more of these determination materials. Specifically, a table as shown in FIG. 4 is prepared, and lighting environments are classified into categories. That is, the lighting environment determination unit 501 observes the determination materials as described above when performing face collation, and outputs which category the current lighting environment belongs to.

The collation performance recording unit 502 records the number of successful face collations and the number of face collations for each predetermined category in the format shown in FIG. When the information input unit 4 obtains an answer to face matching from the user, it records whether or not the face image succeeded in the corresponding category and the number of face matching performed.

The lighting environment comparison unit 503 searches the collation performance recording unit 502 for collation performance in the lighting environment obtained from the illumination environment determination unit 501, and the total number of times of collation (t (K)) is preset. Greater than or equal to the threshold and 2. When the matching performance value obtained by dividing the cumulative number of times that the face matching result is correct (r (K)) by the cumulative number of matching times (t (K)) is greater than or equal to a preset threshold value, Judgment result in lighting environment is judged to be highly reliable. In other cases [when (t (K)) is less than or equal to a preset threshold value, or t (K) is greater than or equal to a threshold value and r (K) / t (K) is less than the threshold value), matching in that environment The result is judged to be unreliable.

(Description of operation)
Next, the operation procedure of this embodiment will be described with reference to the flowchart shown in FIG. In the second embodiment, first, when the control unit 7 determines a transition from the autonomous behavior state to a state for identifying a face (step S601 / YES), the illumination environment determination unit 501 tries to perform face matching. The lighting environment at the time is determined from information from the imaging means 1 and the robot state observation means 6 and is categorized as shown in FIG. 4 as described above (step S602). This category is called the lighting category. An ID is assigned to the illumination category, and the ID is referred to as an illumination ID. Let the ID of the lighting category determined here be K.

Next, the lighting environment comparison unit 503 searches the collation performance recording unit 502 for collation performance when the illumination ID determined by the illumination environment determination unit 501 is K, and the number of times of collation (t (K)). 1. The cumulative total is equal to or greater than a preset threshold, and If the collation performance value obtained by dividing the total number of correct face matching results (r (K)) by the total number of collation times (t (K)) is equal to or greater than a preset threshold, the environment It is determined that the collation result in is highly reliable (step S603 / YES). In other cases [when (t (K)) is less than or equal to a preset threshold value, or t (K) is greater than or equal to a threshold value and r (K) / t (K) is less than the threshold value), matching in that environment It is determined that the result is low in reliability (step S603 / NO). When t (K) is small, face matching is unconditionally performed, and the number of times of matching (t (K)) is incremented, and the number of correct answers (r (K)) is incremented when matching is correct. You may let them.

When the lighting environment comparison unit 503 determines that the collation reliability is high (step S603 / YES), the face detection collation unit 2 detects the face image from the image obtained by the imaging unit 1 and is registered in advance. Collation with the face image is performed, and correspondence is made with someone who is registered (step S604). Let the resulting person be L. Thereafter, the information notification means 3 informs the user of the collation result and asks the user to confirm whether the collation result is correct (step S605). In response to this, the information input means 4 acquires the answer indicated by the user, and determines whether the answer is correct or incorrect (step S606). When the answer is correct (step S606 / YES), the number t (K) of verification when the illumination ID is K and the number r (K) of correct answer when the illumination ID is K are incremented by one. Return to autonomous behavior such as dialogue (steps S609 and S607). If the answer is incorrect, the number of collations t (K) is incremented by 1 (step S607), and the process returns to autonomous behavior such as dialogue with the user.

Next, when the lighting environment comparison unit 503 determines that the verification reliability is low (NO in step S603), the robot body is moved using the traveling unit 8 without performing the face verification (step S608), and again. Observe and determine the lighting environment. If the condition in step S603 is not satisfied even if the movement is performed many times, an implementation method that provides an operation flow that performs face matching is also conceivable. It should be noted that it is desirable to give a natural effect on the movement of the robot so as not to give the user a sense of incongruity.

As described above, every time face matching is performed, the present embodiment performs observation of the lighting environment at that time and checks whether the matching result is correct, and determines the number of times the matching was performed in the lighting environment and the number of correct answers. Cumulatively record. If the accumulation is sufficient, observe the lighting environment at the time of collation, check the percentage of correct answers in the past in the lighting environment at that time, and if there are few correct answers, the robot without collation Move to change its position and direction. In this way, collation errors can be reduced by avoiding collation in a collation environment in which the collation result is not good by looking at past collation histories.

There are various variations in lighting environments and face registration states in different homes, and it is difficult to design a robot by accurately predicting in advance in which lighting environment matching will be successful. Therefore, when the user actually learns what kind of place and in what lighting state the collation is not successful, the collation can be performed with higher accuracy than when assuming poor lighting conditions in advance. it can.

(Third embodiment)
(Description of configuration)
A third embodiment according to the present invention will be described in detail with reference to the drawings. Referring to FIG. 7, the third embodiment according to the present invention includes an overall feature amount extraction unit 504 that extracts an entire feature amount of a person in addition to the second embodiment.

The overall feature quantity extraction unit 504 is controlled by the control unit 7 and extracts a feature quantity that can identify a person from the image captured by the imaging unit 1. The entire feature amount extraction unit 504 includes a dictionary in which feature amounts for each user are registered in advance, and extracts a feature amount that identifies the user from the image captured by the imaging unit 1. The feature amount registered in the dictionary is information that enables the user to be identified, for example, whether the user's height, body shape, hairstyle, or glasses. The whole feature quantity extraction unit 504 displays information indicating that there is a feature quantity when the whole feature quantity can be extracted, and information indicating that there is no feature quantity when the whole feature quantity cannot be extracted. The data is output to 503.

The illumination environment comparison unit 503 changes the threshold for determining the collation performance in the illumination category determined by the illumination environment determination unit 501 depending on whether or not the entire feature amount is extracted by the entire feature amount extraction unit 504. Determine.

Next, the operation procedure of this embodiment will be described with reference to the flowchart shown in FIG. First, when the control unit 7 determines a transition from the autonomous behavior state to a state for identifying a face (step S801), the imaging environment 1 indicates the illumination environment when the collation environment determination unit 501 attempts to perform face collation. Or from the information from the robot state observation means 6 and categorize into illumination categories (step S802). It is assumed that the illumination category ID determined here is K.

The overall feature quantity extraction unit 504 attempts to extract a feature quantity that can identify the user from the image captured by the imaging unit 1 while referring to the dictionary (step S803). If the feature amount can be extracted (step S803 / YES), information indicating the presence of the feature amount is output to the illumination environment comparison unit 503. If the feature amount cannot be extracted (in step S803 /), information indicating no feature amount is output to the illumination environment comparison unit 503.

When the illumination environment comparison unit 503 acquires the information on the illumination ID from the illumination environment determination unit 501 and also acquires the information on the presence / absence of the entire feature amount from the entire feature amount extraction unit 504, the illumination ID acquired from the illumination environment determination unit 501 The collation performance recording means 502 is searched for collation performance when is K.

If the overall feature quantity extraction unit 504 cannot extract the overall feature quantity, the lighting environment comparison unit 503 indicates the total number of times that the face matching result is correct (r (K)). The verification performance value divided by the cumulative total (t (K)) is compared with the threshold value (referred to as R1) used in the second embodiment described above (step S804). If the entire feature amount can be extracted, the comparison performance value is compared with a threshold value (referred to as R2) set lower than this threshold value (step S805).

When the total number of collations is equal to or greater than a threshold and the collation performance value is equal to or greater than a preset threshold, the illumination environment comparison unit 503 determines that the collation result in that environment is highly reliable. In other cases, it is determined that the collation performance in the collation environment is low in reliability. The following operation procedure is the same as the operation procedure of the second embodiment described with reference to FIG.

As described above, according to the present embodiment, it is possible to perform the user's determination with higher accuracy by introducing not only the illumination environment but also the feature amount specifying the user into the material for determining the user.

(Fourth embodiment)
(Description of configuration)
A fourth embodiment according to the present invention will be described in detail with reference to the drawings. Referring to FIG. 9, the fourth embodiment according to the present invention includes a travel plan generation means 9 for making a travel plan for moving to an appropriate position for verification in addition to the configuration of the second embodiment.

In the present embodiment, as shown in FIG. 10, one room is divided into several areas, and an identification ID is added to each divided area for management. Moreover, the collation performance recording means 502 records and manages the illumination category and the collation performance in the illumination category for each region to which the identification ID is attached as shown in FIG.

Further, the robot state observing unit 6 of the present embodiment acquires the state information of the lighting device from the lighting unit state observing unit (not shown), and also acquires the curtain state information by the curtain state observing unit (not shown). The lighting device state observation means detects whether the lighting device is turned on in conjunction with the lighting device installed in the room. The curtain state observation means detects whether the curtain is open in conjunction with the curtain.

The travel plan generation unit 9 acquires the current robot position information from the robot state observation unit 6, searches the collation performance recording unit 502 for collation performance, and moves the collation performance value to a position higher than the current position. In this way, a travel plan is prepared and an instruction is given to the travel means 8. Specifically, in the lighting category in which the time, curtain status, and lighting device status are the same as the current status, the verification performance value obtained by dividing the total number of correct facial verification results by the total number of verifications Is instructed to move to a position above or above the threshold at or above the current lighting category.

In addition, the travel plan generation unit 9 may determine the destination in consideration of the collation performance value and the movement distance. In this case, the collation performance in the same collation category as the current lighting category is higher than that in the current position, so that the movement distance calculated from the current position information acquired from the robot state observation means 6 moves to a short region. A movement plan will be made.

As described above, in the present embodiment, the collation performance value is moved to a position higher than or equal to the threshold value or the highest position in the same lighting category where the time, the curtain state, and the lighting device state are the same as the current state. , Can lead the user to a good lighting environment.

(Fifth embodiment)
(Description of configuration)
A fifth embodiment according to the present invention will be described in detail with reference to the drawings. Referring to FIG. 12, the illumination environment determination unit 500 according to the fifth embodiment of the present invention determines the illumination environment based on the image information obtained from the imaging unit 1 and the time and position information obtained from the robot state observation unit 6. Lighting environment determination unit 501 that determines and corresponds to a predetermined category, and lighting environment record at the time of registration that stores, for each user, an illumination ID when a face image is registered in the face detection / collation unit 2 Based on the information recorded in the means 10 and the lighting environment recording means 10 at the time of registration, the lighting environment comparison means 11 at the time of registration determines whether or not the user during the conversation has registered a face image in the past in the current lighting environment. And.

The lighting environment recording unit 10 at the time of registration records, for each user, all the lighting IDs determined by the lighting environment determination unit 501 when the user registers a face in the face detection / collation unit 2.

The registration lighting environment condition comparison unit 11 records in the registration lighting environment recording unit 10 whether or not the user currently interacting has registered a face image in the past in the current lighting environment obtained from the lighting environment determination unit 501. It is determined by searching for information.

(Description of operation)
Next, the operation procedure of this embodiment will be described with reference to the flowchart shown in FIG. The present embodiment relates to a method for registering a face in the face detection / collation means 2. When the robot is interacting with the user L, a transition is made to a state in which it is determined whether or not to perform registration with a certain probability (step S1301). First, the process branches depending on whether or not the face image of the interacting user is already registered in the face detection / collation unit 2 (step S1302). If not registered (step S1302 / NO), unconditionally go to step S1304, if registered (step S1302 / YES), go to step S1304 with a probability of 1/10, probability of 9/10 To return to user interaction. Here, a natural number from 1 to 10 is randomly generated, and when the generated value is 1 (step S1303 / YES), the process proceeds to step S1304. In step S1304, the lighting environment determination unit 501 classifies the lighting environment at that time into predetermined lighting categories. Let K be the illumination ID of the resulting category. In step S1305, it is determined based on the information recorded in the lighting environment recording unit 10 at the time of registration whether or not the user L who is talking has registered a face in the environment where the lighting ID is K in the past. If the face image when the lighting ID of the user L is K is registered in the lighting environment recording means 10 at the time of registration (step S1305 / YES), the process returns to the dialogue with the user and is not registered. (Step S1305 / NO), the process proceeds to step S1306. In step S1306, face registration is performed by the face detection collating unit 2, and the fact that the user L has registered the face with the illumination ID = K is recorded in the illumination environment recording unit 10 at the time of registration (step S1307).

As described above, the present embodiment stores the lighting state when each user is registered, and the lighting environment in which the user has not registered the face in the past while the user is interacting with the robot. If there is, the user's face registration is started. By doing so, it is possible to register face images under various illumination conditions more efficiently.

In the above-described embodiment, the dialogue with the user and the face matching are described as independent actions, but a face image may be taken during the dialogue.

(Sixth embodiment)
(Description of configuration)
A sixth embodiment of the present invention will be described in detail with reference to the drawings. Referring to FIG. 14, the illumination environment determination unit 500 according to the present embodiment includes only the illumination environment determination unit 501. Further, the face detection collating unit 2 detects a face area from the image obtained by the imaging unit 1 and extracts the face region and converts it into feature data. The face feature data group for each person is determined in the illumination environment. Data management means 203 for storing for each lighting category obtained by means 501, dictionary creation means 204 for creating a recognition dictionary with feature data for each lighting category, and recognition dictionary created by the dictionary creation means 204 for each lighting category A dictionary storage unit 205 to store, a dictionary having the same lighting category as the output of the lighting environment determination unit 501 is selected from the dictionary storage unit 205 and supplied to the face matching unit 202, and supplied from the dictionary selection unit 206. A face collating unit 202 that collates with which face the feature data supplied from the face detecting unit 201 is based on the dictionary.

The data management unit 203 classifies and stores the facial feature data group for each person acquired at the time of registration according to the illumination category obtained by the illumination environment determination unit 501 as shown in FIG. The feature data is extracted from the face image by scanning the front face image from left to right line by line, and when one line ends from top to bottom, the next line is scanned to generate one-dimensional data (raster Scan) and use it as feature data. In addition, a method may be used in which filtering is performed using a first-order differential filter or a second-order differential filter, and edge information is extracted and raster scanned to obtain feature data.

The dictionary creation means 204 creates a recognition dictionary using only feature data of the same lighting category. In FIG. 16, a dictionary is created using each data set with illumination IDs T1 and T2. As the dictionary, it is possible to use a linear discrimination dictionary or the like generated from person feature data described in JP-A-2000-222576. These dictionaries are also stored in the dictionary storage means 205 for each lighting category.

The dictionary selection unit 206 supplies the face collation unit 202 with a dictionary created from the data of the lighting category determined by the lighting environment determination unit 501. In the example of FIG. 17, the dictionary at T2 stored in the dictionary storage unit 205 is supplied.

The face collating unit 202 collates with the feature data supplied from the face detecting unit 201 using the supplied dictionary. Specifically, the pattern identification method is applied by obtaining the degree of similarity with the target face image using a linear discrimination dictionary created from human feature data disclosed in Japanese Patent Laid-Open No. 2000-222576. be able to.

(Description of operation)
Next, the operation procedure of this embodiment will be described with reference to the flowcharts shown in FIGS. First, the operation at the time of face registration will be described with reference to FIG. When the robot is interacting with the user L, a transition is made to a state in which it is determined whether or not to perform registration with a certain probability (step S1801). First, the process branches depending on whether or not the face image of the interacting user is already registered in the face detection / collation unit 2 (step S1802). If not registered (step S1802 / NO), unconditionally go to step S1804. If registered (step S1802 / YES), go to step S1801 with a probability of 1/10, probability of 9/10 To return to user interaction. Here, a natural number from 1 to 10 is randomly generated. If the generated value is 1 (step S1803 / YES), the process proceeds to step S1804. In step S1804, the lighting environment determination unit 501 classifies the lighting environment at that time into predetermined lighting categories. Let K be the illumination ID of the resulting category. In step S 1805, image capturing is started using the imaging unit 1, facial feature data is acquired from the image by the face detection unit 201, and the acquired data is stored in the “user L, lighting category” of the data management unit 203. Store in the area of “K”. In general, a plurality of feature data are acquired even with one registration, and this is continued until the acquisition is completed. When the acquisition is completed, the process proceeds to step S1806, where the dictionary creation unit 204 creates a dictionary for the data with the illumination ID K and stores it in the dictionary storage unit 205. At this time, if not all the data with the illumination ID K are available, the process is interrupted. Or in that case, the operation | movement of creating a dictionary using all the data irrespective of a lighting category may be sufficient.

Next, the operation during face matching will be described with reference to FIG. First, when a transition to a state for identifying a face is determined by the control means 7 (step S1901), the illumination environment when the face matching is to be performed by the illumination environment determination means 501 is indicated by the imaging means 1 or robot state observation means. Judging from the information from No. 6, it is classified into predetermined lighting categories. The resulting illumination ID is K (step S1902). Next, the dictionary selection unit 206 checks whether the dictionary created by the feature data with the illumination ID K is stored in the dictionary storage unit 205 (step S1903). If not saved (step S1903 / NO), the collation is stopped as it is and the process returns to the dialog with the user. If stored (step S1903 / YES), the dictionary selected by the dictionary selection unit 206 using the feature data with the verification ID K is acquired from the dictionary storage unit 205 and supplied to the face verification unit 202 (step S1904). Next, imaging of the image is started by the imaging unit 1, face feature data is acquired from the image by the face detection unit 201, and collation is performed by the face matching unit 202 using the acquired feature data and the supplied dictionary. Then, it is determined who the face image imaged by the imaging means 1 is (step S1905).

As described above, according to the present embodiment, face images are registered for each of various illumination environments, and when collation is performed, collation is performed using a dictionary created from images registered in the same illumination environment as that time. . By doing so, collation can be performed with high accuracy.

Also, in the present embodiment, the dialogue with the user and the face matching are described as independent actions, but the matching may be performed by taking a face image during the dialogue.

The above-described embodiment is a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention. For example, as a material for determining the verification environment in the above-described embodiment, the season (because the angle at which sunlight enters the room differs between summer and winter) and the weather forecast (we obtain the weather information of the day using the Internet etc. One of the materials for determining the environment) may be considered.

Further, in the embodiment according to the collation environment determination program of the present invention, each means performs the control operation based on the program controlled by the operation procedure shown in FIGS. 2, 6, 8, 13, 18, and 19, respectively. Realized.

1 is a block diagram showing a configuration of a first embodiment according to the present invention. It is a flowchart which shows the operation | movement procedure of 1st Embodiment which concerns on this invention. It is a block diagram which shows the 2nd Embodiment which concerns on this invention. It is a figure which shows the method of classifying an illumination environment by the illumination environment judgment means in 2nd Embodiment. It is a figure which shows the method of recording the log | history of a collation result for every lighting environment with the collation performance recording means in 2nd Embodiment. It is a flowchart which shows the operation | movement procedure of 2nd Embodiment which concerns on this invention. It is a block diagram which shows the structure of 3rd Embodiment based on this invention. It is a flowchart which shows the operation | movement procedure of 3rd Embodiment which concerns on this invention. It is a block diagram which shows the structure of 4th Embodiment based on this invention. It is a figure for demonstrating the method which divides | segments a room into an area | region and attaches ID for every area | region and manages. It is a figure which shows the method of classifying an illumination environment by the illumination environment judgment means in 4th Embodiment. It is a block diagram which shows the structure of 5th Embodiment based on this invention. It is a flowchart which shows the operation | movement procedure of 5th Embodiment concerning this invention. It is a block diagram which shows the structure of the 6th Embodiment concerning this invention. It is a figure which shows the example which preserve | saves the feature data group of the face for every person acquired at the time of registration for every illumination category with the data management means in 6th Embodiment. It is a figure which shows the example which produces a recognition dictionary using only the feature data of the same illumination category with the dictionary creation means in 6th Embodiment. It is a figure which shows the example which supplies the dictionary created with the data of a certain illumination category to the collation means by the dictionary selection means in 6th Embodiment. It is a flowchart which shows the operation | movement procedure (at the time of face registration) of 6th Embodiment. It is a flowchart which shows the operation | movement procedure (at the time of face collation) of 6th Embodiment. It is a block diagram which shows the structure of the conventional robot apparatus with a face collation function.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging means 2 Face detection collation means 201 Face detection means 202 Face collation means 203 Data management means 204 Dictionary creation means 205 Dictionary storage means 206 Dictionary selection means 3 Information notification means 4 Information input means 5 Collation result reliability calculation means 500 Illumination environment Determination means 501 Illumination environment determination means 502 Collation performance recording means 503 Illumination environment comparison means 504 Overall feature quantity extraction means 6 Robot state observation means 7 Control means 8 Travel means 9 Travel plan generation means 10 Registration illumination environment recording means 11 Illumination at registration Environmental comparison means

Claims (27)

A robot apparatus comprising face detection collation means for collating a person using face image information with respect to an input image,
Lighting environment state observation means for detecting the lighting environment state;
Determining means for determining whether the lighting environment state is suitable for verification ;
Traveling means for moving the robot device ,
Determining whether the lighting environment state is suitable for collation by referring to the collation performance recording unit that records the success or failure of the collation for the lighting environment state;
When it is determined that the lighting environment state is not suitable for collation, collation in the face detection collation unit is not performed ,
In the verification performance recording means, the success / failure of the verification for the positional information and the lighting environment state is recorded,
When the lighting unit determines that the lighting environment state is not suitable for collation, the robot device moves to a place suitable for collation by the traveling unit . The robot apparatus according to claim 1 , wherein when the determination unit determines that the lighting environment state is not suitable for verification, the traveling unit moves to a location where verification is successful among the locations. . The traveling unit, when the illumination environment state is determined to be unsuitable for verification by the determination means according to claim 1, wherein the moving it to high successful frequency matching of the location Robotic device. The traveling means moves to a place where the frequency of successful matching is higher than the current place among the places when the judging means judges that the lighting environment state is not suitable for the matching. The robot apparatus according to claim 1 . The determination means includes
The variance of the pixel values of the image of the face area detected by the face detection collating means, the ratio of the average brightness value of the face area detected by the face detection collating means and the average brightness value of the outer neighboring area, the average luminance value of the entire image, a robot apparatus according to the determined decisions from claim 1, wherein in any one of 4 to at least one lighting environmental conditions. Robot state observing means for observing the lighting environment state of the room using the illuminating device state observing means for detecting whether the lighting of the room is lit in conjunction with the lighting device of the room,
The determining means is a robot apparatus according to any one of claims 1 to 4, characterized in that one of the reference amount at the time of the categorization of the lighting environmental conditions whether the illumination is attached the room . Robot state observing means for observing the open / closed state of the room curtain using curtain state observing means for detecting whether the room curtain is open in conjunction with the room curtain,
Determining means, robotic device according to any one of claims 1 to 4, characterized in that one of the reference amount when the categorization assimilation lighting environmental conditions curtain the room is open . Robot state observation means that holds information about time,
5. The robot apparatus according to claim 1, wherein the determination unit sets a time when the comparison is performed as one of reference amounts for categorizing the lighting environment state. 6. . Observe the position of the robot device in the room,
It has robot state observation means to hold as position information,
5. The determination unit according to claim 1, wherein the position in the room of the robot apparatus when the collation is to be performed is one of reference amounts for categorizing the lighting environment state. The robot apparatus according to item 1. A collation environment determination method of a robot apparatus having a face detection collation process for collating a person using face image information with respect to an input image,
A lighting environment state observation process for detecting a lighting environment state;
A determination step of determining whether the lighting environment state is suitable for verification ;
A moving step of moving the robot device by a traveling means ,
Determining whether the lighting environment state is suitable for collation by referring to collation performance recording means that records the success or failure of the collation for the lighting environment state in the determination step;
When it is determined that the lighting environment state is not suitable for collation, the collation in the face detection collation process is not performed ,
In the verification performance recording means, the success / failure of the verification for the positional information and the lighting environment state is recorded,
A collation environment determination method , wherein when the lighting environment state is determined to be unsuitable for collation in the determination step, the moving step is moved to a location suitable for collation. The collation environment according to claim 10 , wherein, in the moving step, if the lighting environment state is determined to be unsuitable for collation in the determination step, the collation environment is moved to a location where collation is successful among the places. Judgment method. In the moving step, when the illumination environment state is determined to be unsuitable for verification in the determination step, according to claim 10, wherein the moving it to high successful frequency matching of the location Collation environment judgment method. In the movement step, when it is determined in the determination step that the lighting environment state is not suitable for verification, the location is moved to a location where the verification has been successful more frequently than the current location. The collation environment determination method according to claim 10 . The determination step includes
The variance of the pixel values of the image of the face area detected in the face detection collation process, the ratio of the average brightness value of the face area detected by the face detection collation means and the average of the brightness values of the outer neighboring areas, The collation environment determination method according to any one of claims 10 to 13 , wherein at least one of an average of luminance values of the entire image is used as a determination material for determining an illumination environment state. A robot state observation step of observing the lighting environment state of the room using the lighting device state observation means for detecting whether the lighting of the room is lit in conjunction with the lighting device of the room,
The collation environment according to any one of claims 10 to 13 , wherein in the determination step, one of the reference amounts for categorizing the lighting environment state is whether or not the room is illuminated. Judgment method. A robot state observing step for observing the open / closed state of the room curtain using curtain state observing means for detecting whether the room curtain is open in conjunction with the room curtain;
The collation according to any one of claims 10 to 13 , wherein in the determination step, one of the reference amounts for categorizing the lighting environment state is whether or not the curtain of the room is open. Environmental judgment method. The collation environment according to any one of claims 10 to 13 , wherein, in the determination step, a time when collation is to be performed is set as one of reference amounts for categorizing the illumination environment state. Judgment method. A robot state observation process for observing the position of the robot apparatus in the room,
The determination step, any one of claims 10 to 13, characterized in that one of the reference amount when the position in the room of the robotic device to the categorization of the lighting environmental conditions at the time of attempted verification 2. The verification environment determination method according to item 1. A collation environment determination program for a robot apparatus having face detection collation processing for collating a person using face image information with respect to an input image,
Lighting environment state observation processing for detecting the lighting environment state;
A determination process for determining whether the lighting environment state is suitable for verification ;
And a moving process for moving the robot device by a traveling means ,
Determining whether the lighting environment state is suitable for collation by referring to collation performance recording means in which the judgment process records the success or failure of the collation for the lighting environment state;
When it is determined that the lighting environment state is not suitable for collation, the collation in the face detection collation process is not performed ,
In the verification performance recording means, the success / failure of the verification for the positional information and the lighting environment state is recorded,
A collation environment determination program characterized in that, when it is determined in the determination process that the lighting environment state is not suitable for collation, the movement process moves to a place suitable for collation. The collation environment according to claim 19 , wherein, in the movement process, when the lighting environment state is determined to be unsuitable for collation in the determination process, the collation environment is moved to a place where the collation is successful among the places. Judgment program. In the moving process, when the illumination environment state is determined to be unsuitable for verification in the determination process, according to claim 19, wherein the moving it to high successful frequency matching of the location Collation environment judgment program. In the movement process, when it is determined in the determination process that the lighting environment state is not suitable for collation, the location is moved to a place where the collation succeeded more frequently than the current place. The verification environment determination program according to claim 19 . The determination process includes
The variance of the pixel values of the image of the face area detected by the face detection verification process, the ratio of the average brightness value of the face area detected by the face detection verification means and the average brightness value of the outer neighboring area, The collation environment determination program according to any one of claims 19 to 22 , wherein at least one of an average of luminance values of the entire image is used as a determination material for determining an illumination environment state. A robot state observation process for observing the lighting environment state of the room using the lighting device state observation means for detecting whether the lighting of the room is lit in conjunction with the lighting device of the room,
23. The collation environment according to any one of claims 19 to 22 , wherein the determination processing uses one of reference amounts when categorizing the lighting environment state based on whether or not the room is illuminated. Judgment program. A robot state observation process for observing the open / closed state of the room curtain using curtain state observation means for detecting whether the curtain of the room is open in conjunction with the curtain of the room;
The collation according to any one of claims 19 to 22 , wherein in the determination process, one of the reference amounts for categorizing the lighting environment state is whether or not the curtain of the room is open. Environmental judgment program. The collation environment according to any one of claims 19 to 22 , wherein in the determination process, a time when the collation is to be performed is set as one of reference amounts for categorizing the lighting environment state. Judgment program. It has a robot state observation process to observe the position of the robot device in the room,
The determination process, any one of claims 19, characterized in that one of the reference amount when the position in the room of the robotic device to the categorization of the lighting environmental conditions at the time of attempted verification 22 The collation environment determination program according to item 1.

Images