JP2006179023A - Robot device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a robot device capable of identifying a person even under environment such as family environment where illumination conditions are not fixed. <P>SOLUTION: The robot device has a personal identifier having a video image acquisition means for acquiring an image, a head detection and tracking means for detecting the head of a human being from the image, a front facial positioning matching means for acquiring a front facial image from the inside of a partial image of the detected head, a facial characteristic extraction means for converting the front facial image into featured values, a facial identification means for identifying a person from the featured values using an identification dictionary and an identification dictionary storage means for storing the identification dictionary, a whole control part which controls operations of a robot, a voice output means for uttering voice by an instruction of the whole control part and a transfer means for transferring the robot by the instruction of the whole control part. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for identifying a person in a video, and more particularly to a person identification technique using a front face and a robot apparatus for identifying a person.

  Several methods for identifying a person using facial images have been proposed. Recent trends in face detection and identification technologies are described in, for example, Reference (1) (Shigeru Akamatsu, “Face Recognition by Computer-Survey”, IEICE Transactions, Vol. J80-D-II, No. 8, pp.2031-2046, August 1997). In general, a face identification system includes a process for detecting a face from an image, a feature extraction process from a face pattern, and a person identification process for comparing feature quantities with dictionary data.

  As a detection method of face images, reference (2) (Shin Kosugi, “Finding and locating faces in scenes using multiple pyramids for personal identification”, IEICE Transactions, Vol. J77-D- II, No.4, pp.672-681, April 1994), which performs template matching using shading patterns, and literature (3) (M. Turk, A. Pentland, “Face Recognition”). on Eigenfaces ”, Proceedings of IEEE, CVPR91), a face image eigenvector projection distance method is known.

  Also, for example, Japanese Patent Laid-Open No. 9-251534 has proposed a method of detecting features such as eyes, nose, and mouth and cutting out a front face shading pattern from the positional relationship.

  As a typical example of face detection, an eigenvector projection distance method by M. Turk et al. Will be described.

  Prepare a lot of front face data (hundreds) beforehand. Using these pixel values as feature vectors, eigenvalues and eigenvectors are obtained. P eigenvectors Vn (n = 1,... P) are obtained in descending order of eigenvalues.

  When the test image t is projected onto the eigenvector Vn, p projection values are obtained. A reconstructed test image t 'is obtained by reconstructing the test image from these projection values and the eigenvector Vn.

  If t is close to the face pattern, an image close to the face pattern is obtained as the reconstructed test image t ′. Therefore, it is determined whether or not the face is based on the distance scale Dt given by the following equation (1).

  There are two types of face identification features, one that uses geometric features of facial features such as eyes, nose, and mouth, and one that uses global shade pattern matching. Since the positional relationship of the feature changes as the face orientation and facial expression change, the latter method using the global shading pattern is currently the mainstream.

  Examples of face image identification and collation methods include, for example, the above document (2) (Shin Kosugi, “Finding and locating faces in scenes using multiple pyramids for personal identification”, IEICE Transactions, Vol. J77-D-II, No. 4, pp. 672-681, April 1994) considers a gray pattern as a feature vector, and uses a category with a large inner product between feature vectors as an identification result. Also, in the above document (3) (M. Turk, A. Pentland, “Face Recognition on Using Eigenfaces”, Proceedings of IEEE, CVPR91), the projection value onto the eigenvector of the face image is used as a feature vector, and the Euclidean distance The small category is the identification result.

  Conventionally, as a robot apparatus having an image recognition function, for example, there is an apparatus described in Japanese Patent Application No. 10-151591. This robot apparatus can extract color information from an image and change the operation according to the color pattern. However, no function means for recognizing a person is provided.

  The above-described conventional system has the following problems.

  The first problem is that a person cannot be identified in an environment where lighting conditions are not constant, such as a home environment.

  This is because it is difficult to detect a face in a general environment. For example, the template matching method is difficult to detect unless the face pattern and the dictionary pattern in the image are almost density values, and the illumination direction is slightly shifted or is different from the person in the dictionary. Almost undetectable. On the other hand, although the eigenvector projection distance method has a higher detection performance than the template matching, the detection fails similarly in an image with a different illumination direction or a complicated background.

  Another reason why the person cannot be identified in an environment where the lighting conditions are not constant is that the conventional feature extraction method and the identification method cannot absorb the variation of the feature amount due to the illumination variation.

  Accordingly, the present invention has been made in view of the above problems, and an object thereof is to provide a robot apparatus that can identify a person in a general environment such as a home environment.

  Accordingly, another object of the present invention is to provide a robot apparatus that can stably identify a person in a general environment.

  The robot apparatus according to the present invention that achieves the above-described object includes, as a person identification device, a video acquisition unit that acquires an image, a head detection tracking unit that detects a human head from the image, and a detected head. Front face alignment means for acquiring a front face image from a partial image, face feature extraction means for converting the front face image into a feature quantity, face identification means for identifying a person from the feature quantity using an identification dictionary, and identification An identification dictionary storage means for storing a dictionary is provided. Then, in the head detection and tracking means, a monocular head rectangular coordinate detection means for detecting the head from one image, and a facing distance evaluation means for removing erroneous head detection from the facing distance value and the head rectangular coordinate value And measuring the distance between the front object, a general control unit that controls the operation of the robot, a speaker that speaks voice according to instructions from the general control unit, a moving means that moves the robot according to instructions from the general control unit A face-to-face distance sensor, a touch sensor, a microphone, and voice recognition means.

  In the present invention, when the person identification result is obtained, the overall control unit controls to speak with a different voice for each person.

  In the present invention, when the overall control unit obtains a face-to-face distance and direction with a front object from the person identification device, means for obtaining a person identification result, and the face-to-face distance is equal to or greater than a threshold value And means for moving so as to approach the forward object, and means for controlling the person identification result to be spoken with a different voice for each person when the facing distance is less than or equal to a threshold value.

  As described above, according to the present invention, the following effects can be obtained.

  The person identification device used in the robot apparatus of the present invention has an effect that a person can be identified with an extremely high identification rate even in an environment where the lighting conditions fluctuate significantly, such as a home environment.

  The reason is as follows. That is, in the present invention, detection and identification are performed using not only the gray value of the image but also the motion and the facing distance information. Further, in the present invention, the front face with high accuracy is detected by the face alignment means. Furthermore, in the present invention, the face identification means performs similarity identification using a linear discrimination dictionary.

  And in this invention, it is because it comprised so that an identification precision could be raised by an apparatus or a system learning by itself through the dialogue with a person (user).

  Also, according to the robot apparatus of the present invention, even if the input image is not good and cannot be identified at a certain place, a good image can be obtained by operating the apparatus itself thereafter.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing a configuration of an embodiment of a person identification device used in a robot apparatus according to the present invention. Referring to FIG. 1, a person identification device 14 according to an embodiment of the present invention includes a video acquisition unit 2, a face-to-face distance sensor 5, a person detection / identification unit 1, a person detection / identification management unit 13, and dictionary data. And a management unit 12. The video acquisition means 2 includes a right camera 3 and a left camera 4 and acquires respective camera video information. The facing distance sensor 5 is installed in the same direction as the optical axis of the camera, and measures the facing distance to an object in the video. Examples of the facing distance sensor 5 include an ultrasonic sensor and an infrared sensor. The person detection / identification management unit 13 transmits an operation start command and an operation end command to the person detection / identification unit 1, transmits characteristic data to the dictionary data management unit 12, and transmits a dictionary creation command.

  Cameras used in an embodiment of the present invention collectively include, for example, video devices, digital CCD cameras, and the like, and photographing devices that can output a moving scene as a sequence of still images.

  The person detection and identification unit 1 includes a head detection and tracking unit 6, a front face alignment unit 7, a face feature extraction unit 8, a face identification unit 9, an identification dictionary storage unit 10, and an identification result correction unit 11. I have.

  Upon receiving the operation start command from the person detection / identification management unit 13, the person detection / identification unit 1 starts the operation after loading the identification dictionary from the dictionary data management unit 12 into the identification dictionary storage unit 10.

  FIG. 18 is a flowchart for explaining the processing of the person detection / identification means 1 according to the embodiment of the present invention. The operation of the person detection / identification means 1 will be described with reference to FIGS.

  First, the head detection and tracking unit 6 outputs the number of human heads in the current frame and head rectangular coordinates based on the image information from the video acquisition unit 2 and the reading value of the facing distance sensor 5. (Step S1).

  Next, the number of detected heads is evaluated (step S2). If the detected number of heads is 0, the next frame image is input to detect heads, and step S1 is continued until the detected number becomes 1 or more.

  When the number of detected heads is 1 or more, the detection result is transmitted to the front face alignment means 7. The front face alignment means 7 performs a face area search process (step S3) and determines whether a front face area has been found (step S4).

  When a front face is found, a front face image that is a rectangular image at the center of the face is output. The processes in steps S3 and S4 are intended to eliminate erroneous detection of the head, and to extract only the video in which the person is facing the front of the camera and send it to the subsequent process. If the front face cannot be found, the processing is performed again from step S1.

  When a front face is found, the face feature extraction means 8 converts the front face image into feature amount data (step S5).

  As shown in FIG. 8, one example of the face feature extraction means 8 scans the front face image from left to right line by line, and when one line ends from top to bottom, scans the next line to obtain one-dimensional data. It is generated (called “raster scan”) and used as feature data. In addition, a method in which edge data is extracted by filtering using a first-order differential filter or a second-order differential filter may be used as a feature data by raster scanning.

  Next, the face identification means 9 performs face identification processing with reference to the dictionary data in the identification dictionary storage means 10 (step S6).

  Next, the identification result correction unit 11 performs integration processing with identification results for past m frames (m is an integer of 2 or more) (step S7), and outputs the result to the person detection identification management unit 13 ( Step S8).

  At this time, when a plurality of head rectangles are detected in the head detection / tracking means 6 (step S1) and all of them are not processed (No branch in step S9), the front face alignment means 7 again ( Processing is performed from step S3). The person detection / identification unit 1 ends upon receiving an end instruction from the person identification management unit 13 (step S10). The process is continued again from step S1 until an end instruction is given.

  FIG. 2 is a diagram showing the configuration of the head detection / tracking means 27 constituting an embodiment of the head detection / tracking means 6 of FIG. Referring to FIG. 2, the head detection tracking unit 27 includes a head detection unit 21, a head tracking unit 22, and a head rectangular coordinate storage unit 23. The head detecting unit 21 includes a head rectangular coordinate detecting unit 24, a left / right image collating unit 25, a facing distance integrating unit 31, and a facing distance evaluating unit 26.

  FIG. 19 is a flowchart for explaining the processing of the head detection / tracking means 27. With reference to FIGS. 2 and 19, the operation of the head detection and tracking means 27 according to an embodiment of the present invention will be described.

  The right camera image, the left camera image, and the reading value of the facing distance sensor are input to the head detecting means 21. The head detection means 21 performs a human head detection process from the input information, and outputs head rectangular coordinates and the number of head detections (step S10).

  When the number of detected heads is 1 or more, the number of detected heads and the head rectangular coordinates are stored in the head rectangular coordinate storage means 23 and then output (step S18).

  If the number of detected heads is 0, the head tracking means 22 takes out the head rectangle information in the previous frame from the head rectangle storage means 23 and performs head tracking processing (step S19).

  If head tracking is successful, the number of heads and head rectangular coordinates that have been successfully tracked are output, and if tracking fails, the number of detections is 0 (step S20).

  Next, the operation of the head detecting means 21 in step 10 of FIG. 19 will be described in detail. The head detecting means 21 first inputs either the left or right video to the head rectangular coordinate detecting means 24 to obtain the provisional head detection number and provisional head rectangular coordinates (step S11).

  The head rectangular coordinate detection means 24 shown in FIG. 2 uses a right camera image. Next, the left and right image collating means 25 calculates the facing distance value based on the principle of stereo vision using the obtained head rectangular coordinates and the left and right camera images (step S12 in FIG. 19).

  With reference to FIG. 6, the operation of the left and right image matching means 25 in FIG. 2 will be described. A head rectangle detected in the right camera image is set as a head detection rectangle 51. Then, the vicinity of the same detected coordinate position of the left camera image is searched using the image data in the head detection rectangle 51. An example of the search method is template matching. If the gray value of the right camera image is FR (x, y), the gray value of the left camera image is FL (x, y), the horizontal size of the rectangle is Tw, and the vertical size is Th, the upper left starting point position of the template is The matching distance Dtm when it is at (sx, sy) of the camera image is expressed by the following equation (2).

  The above equation (2) represents the Euclidean distance between the partial images of the right camera and the left camera. The coordinates on the left camera image when Dtm is the smallest are taken as search results 52. When the search result 52 is obtained, next, the left and right rectangular coordinate values are compared, and the distance to the human head is calculated.

With reference to FIG. 32, an example of a method for calculating the distance to the object will be described. FIG. 32 is a top view of a situation where a single target object 403 is captured using the left and right cameras. The right camera 401 and the left camera 402 are installed in parallel with an interval C therebetween. The angle of view of the camera is θ and is the same for both left and right. Let e be the lateral length of the imaging surface of the camera. In this state, the object 403 is shown at the coordinate Xr in the right camera image, and the coordinate Xl is shown in the left camera image. The maximum horizontal size of the image is W pixels. At this time, the facing distance Z from the camera imaging surface to the symmetrical object 403 can be calculated by the following equation (3).

  Here, since e is a small value usually less than 1 cm, it can be approximated as 0. As described above, the facing distance is calculated from the left and right camera images.

  Then, after the face-to-face image collating means 25 calculates the face-to-face distance by stereo vision, the face-to-face distance integrating means 31 performs face-to-face distance integration processing based on the output value of the face-to-face distance sensor 30 (step in FIG. 19). S13). Experimentally, distance sensors such as ultrasonic sensors are very accurate when the distance is less than 1 m. On the other hand, the error tends to increase at a distance of 1 m or more.

  Although the distance value calculated by stereo viewing is effective up to about 3 m depending on the angle of view of the camera, if the distance is too close, the error tends to increase. Therefore, as a method of integrating both distance values, when the output of the facing distance sensor 30 is smaller than a certain threshold value T, the value of the facing distance sensor is adopted, and when the output is larger than the threshold value T, A method of adopting a distance value by stereo vision is used.

  After integrating the face-to-face distances, the face-to-face distance evaluation means 26 calculates the actual size of the head from the face-to-face distance value and the head rectangular coordinate value in the image (step S14 in FIG. 19).

  If the calculation result substantially matches the size of the human head, it is determined that the head is really detected. If the calculation result is significantly different from the actual head size, it is determined that it is a false detection (step S15 in FIG. 19). For example, when the horizontal size of the head is within 12 cm plus or minus 2 cm and the vertical size is within 20 cm plus or minus 4 cm, it is regarded as the head, and otherwise it is determined that it is not the head.

  If it matches the actual size, the number of detections is increased by 1 (step S16 in FIG. 19).

  When the tentative head rectangular coordinates that have not been evaluated remain (No branch in step S17 in FIG. 19), the process is performed again from step S12. The head detecting means 21 outputs the number of detected heads and the head rectangular coordinates when all the provisional head rectangular coordinates have been evaluated (Yes branch in step S19 in FIG. 19).

  Next, the head rectangular coordinate detection means 24 shown in FIG. 2 will be described. FIG. 4 is a diagram showing the configuration of the head rectangular coordinate detecting means 41 constituting an embodiment of the head rectangular coordinate detecting means 24. As shown in FIG. Referring to FIG. 4, the head rectangular coordinate detection unit 41 includes a moving pixel detection unit 42, a noise removal unit 47, a person number evaluation unit 43, a top coordinate detection unit 44, and a head lower coordinate detection unit 45. The temporal coordinate detection means 46 is provided.

  FIG. 20 is a flowchart for explaining the processing of the head rectangular coordinate detection means 41. The operation of the head rectangular coordinate detection means 41 will be described with reference to FIGS. 20, 4, and 5.

  First, the moving pixel detection means 42 detects a pixel group that moves within the screen. The difference between the input image data and the image data input immediately before is taken to generate a difference image g (step S21).

  Furthermore, the difference image g of past m frames (m is an integer of 2 or more) is added and averaged to obtain an integrated difference image G (step S22). The integrated difference image G has a pixel value of 0 in a non-motion area, and a larger pixel value in a movement area.

  Since the integrated difference image G contains a large amount of sesame salt noise, the noise removal unit 47 performs noise removal processing (step S23). Examples of noise removal processing include expansion / contraction processing and median filter processing. These noise removal processes are common in the field of image processing, and processes well known to those skilled in the art are used.

  Next, the number-of-persons evaluation means 43 in FIG. 4 evaluates how many people are on the screen. The operation of the person number evaluation means 43 will be described. FIG. 5 is a diagram for explaining an example of acquisition of the integrated difference image G.

  First, a method for detecting only one person will be described. If the integrated difference image G48 is obtained, it is first determined whether or not there is a motion region (step S24 in FIG. 20). Here, the motion area represents an area occupied by a moving pixel. If there is no motion area, that is, if the integrated difference image G is all zero, the number of persons is determined to be zero. Otherwise, the number of persons is 1.

  Next, a method for detecting a plurality of persons will be described. If the integrated difference image G49 is obtained, first, the presence / absence of a motion region is checked (step S24 in FIG. 20). When there is no motion area, the number of persons is zero. If there is a motion region, the integrated difference image G is referred to determine how many people are present (step S25 in FIG. 20). As a determination method, for example, there is a method in which one person is used when the maximum value of the motion area width in the integrated differential image upper area 50 is smaller than a certain threshold value, and two persons are used when the maximum value is larger. When the number of persons is two, it is assumed that the persons are arranged side by side, and the integrated difference area G is divided into the partial area 1 and the partial area 2. Note that even when three or more people are detected, it can be handled by increasing the number of divisions. When obtaining the head rectangle, the same processing described below (from step S26 to step S29 in FIG. 20) may be repeated for each of the partial area 1 and the partial area 2.

  Next, processing for obtaining head rectangular coordinates from the integrated difference image G will be described. A motion region width 47 is obtained for each scan line (step S26 in FIG. 20).

  The motion region width 47 represents the difference between the maximum and minimum x-coordinate values of the motion region in each scan line.

  Next, the Y coordinate of the top is obtained by the top coordinate detection means 44 (step S27 in FIG. 20). As a method of obtaining the vertex coordinates, there is a method in which the minimum value of the Y coordinate of the motion region is used as the vertex.

  Next, the Y coordinate of the bottom of the head rectangle is obtained by the head lower coordinate detecting means 45 (step S28 in FIG. 20). As a method for obtaining the base coordinates of the head rectangle, a search is performed from the top to the bottom (Y direction), a line whose motion region width 47 is smaller than the average value dm of the motion region width is obtained, and the Y coordinate in the line is obtained. A method may be used in which the largest is the base of the head rectangle.

  Next, the left and right x-coordinates of the head rectangle are obtained by the temporal coordinate detection means 46 (step S29 in FIG. 20). As a method of obtaining the left and right x-coordinates, a method of obtaining the coordinates of the left and right ends of the motion region in the line having the largest motion region width 47 in the range from the top of the head to the lower part of the head may be used.

  When the number of persons is two or more, the processing from step S26 to step S29 in FIG. 20 is repeated for each partial region.

  Next, the operation of the head tracking means 22 in FIG. 2 will be described with reference to FIG. The tracking process is performed on the camera image (right camera image in FIG. 2) used for head rectangular coordinate detection. First, the head rectangular coordinate 53 of the previous frame and the head rectangular image 55 of the previous frame are read from the head rectangular storage means 23.

  Next, in the current frame, the vicinity region of the head rectangular coordinate 53 of the previous frame is searched by template matching, and the place with the smallest distance value is set as the tracking result.

  FIG. 3 is a diagram showing the configuration of the head detection / tracking means 32 according to another embodiment of the head detection / tracking means 6 of FIG. Referring to FIG. 3, the head detection and tracking unit 32 includes a head detection unit 33, a head rectangle storage unit 23, and a head tracking unit 22. 2 is different from the embodiment shown in FIG. 2 in that the head detecting means 33 has a head rectangular coordinate detecting means 24 and a face-to-face distance evaluating means 26, and outputs from the monocular camera 34 and the face-to-face distance sensor 30. The detection is performed using. That is, the head rectangle is evaluated using only the reading value of the facing distance sensor without considering the facing distance in the left and right stereo vision.

  As another example of the head detection and tracking means 6, a head detection means having a configuration in which a face distance is obtained from information of only the left and right cameras and a head rectangle is evaluated without using a face distance sensor. Also good. In the case of this configuration, the head detection unit 21 is configured without the facing distance integration unit 31 of FIG.

  FIG. 9 is a diagram showing the configuration of the front face alignment means 61 constituting one embodiment of the front face alignment means 7 of FIG. Referring to FIG. 9, the front face alignment unit 61 includes a head rectangular cutout unit 62, a front face search unit 63, and a front face appearance determination unit 65.

  The front face appearance determination unit 65 includes a density dispersion determination unit 66 and a threshold processing unit 67.

  FIG. 21 is a flowchart for explaining the processing of the front face alignment means 61. The operation of the front face alignment means 61 will be described with reference to FIGS. When the image data, the head rectangular coordinates, and the facing distance are input, the front face alignment unit 61 outputs a front face presence / absence flag and front face image data. The input image data is cut into a partial image of the head rectangle by the head rectangle cutting means 62 (step S41). This partial image is referred to as a “head rectangular image”.

  Next, the front face search means 63 searches the front face area from the head rectangular image, and outputs the inter-pattern distance or similarity between the front face image and the standard face dictionary (step S42).

  Next, the front face likelihood determination means 65 determines whether or not the front face image is really a front face (step S43). If it is determined that the face is a front face, the front face presence / absence flag is “present” and a front face image is output. If it is determined that the face is not a front face, the front face presence / absence flag is “none” and no front face image is output.

  The front face appearance determination unit 65 includes a density dispersion determination unit 66 and a threshold processing unit 67.

  The density dispersion determining unit 66 obtains the variance of the gray value of the front face image data, and determines that the face is not a front face if it is below a certain threshold value (step S44 in FIG. 21).

  The density dispersion determining means 66 can eliminate a monotonous wall-like pattern.

  The threshold value unit 67 determines the likelihood of a front face by thresholding the inter-pattern distance or the similarity (step S45 in FIG. 21).

  In the case of the inter-pattern distance value, it is determined that the face is not a front face when it is equal to or greater than the threshold value. In the case of similarity, it is determined that the face is not a front face when it is below the threshold.

  FIG. 12 is an explanatory view schematically showing the operation of the front face alignment means 61. If the head rectangle 151 is detected, a head rectangle image 152 is generated in step S41 of FIG.

  Next, in the face center portion search process in step S42 of FIG. 21, the front face image 155 is obtained after the reduced head rectangular image 153 is generated.

  The front face image is an image of the center part of the face as shown in the front face image 155 of FIG. 12, the horizontal direction is such that both eyes are completely included, and the vertical direction is the entire mouth from the eyebrows. It means an image of an area that includes it.

  FIG. 10 is a diagram showing the configuration of the front face searching means 71 that constitutes one embodiment of the front face searching means 63. Referring to FIG. 10, this front face search means 71 includes a head rectangular image storage means 89, a head intermediate size calculation means 88, an image reduction means 90, an intermediate size storage means 91, and a front face candidate extraction means. 72, intermediate reduced image storage means 73, contrast correction means 74, eigenvector projection distance calculation means 75, standard face dictionary storage means 76, storage means 77, projection distance minimum determination means 78, and search range end determination. Means 79 and multi-resolution processing end judging means 92 are provided.

  The eigenvector projection distance calculation means 75 includes an average difference means 82, a vector projection value calculation means 83, a reconstruction calculation means 84, and a projection distance calculation means 85.

  The standard face dictionary storage unit 76 includes a standard face average data storage unit 80 and a standard face eigenvector data storage unit 81.

  The storage unit 77 includes a projection distance minimum value storage unit 86 and a front face gray value storage unit 87.

  FIG. 22 is a flowchart for explaining the process of the front face searching means 71. The operation of the front face searching means 71 will be described with reference to FIGS. The head rectangular image data is held in the head rectangular image storage unit 89. First, the head intermediate size calculating means 88 calculates the intermediate reduced size of the head rectangular image with reference to the face distance value and the size of the standard face dictionary data (step S101).

  A processing example of the head intermediate size calculating unit 88 will be described. The intermediate reduced size is shown as the vertical and horizontal size of the reduced head rectangular image 153 in FIG. The horizontal size of the head rectangular image 152 is Hw, and the vertical size is Hh. The horizontal size of the intermediate reduced size is Mw, and the vertical size is Mh. The front face image has a horizontal size Fw and a vertical size Fh. Fw and Fh are the same as the vertical and horizontal sizes of the front face search shape 154, and are uniquely determined for the standard face dictionary. Hh, Hw, Mh, Mw, Fh, and Fw are all pixel sizes in units of pixels.

  The standard face dictionary is a pattern recognition dictionary generated by using the gray value of the front face area shown in the front face image 155 of FIG. 12 as a feature value. The front face region means a region that includes both eyes completely in the horizontal direction and a region that includes the entire mouth from the eyebrows. The front face area is not necessarily rectangular, and can be realized by an arbitrary continuous area including both eyes, nose and mouth, such as an ellipse. However, if the shape is rectangular, the processing can be simplified and speeded up, which is effective as a mounting form. Therefore, in the following description, the front face area is assumed to be rectangular.

  If the actual vertical and horizontal sizes of the front face area are RFh and RFw, a male adult can be represented by about RFw = 10 cm and RFh = 15 cm. On the other hand, since the facing distance Z is known, the actual vertical and horizontal sizes RHh and RHw of the head rectangular image can be calculated by the following equation (4). The variable of the following equation (4) corresponds to FIG.

  Since the width e of the image pickup surface is small, there is no problem even if the calculation is normally ignored.

  In order to search for a head rectangular image using the standard face dictionary, it is necessary to convert the head rectangular image to the same resolution as the standard face dictionary. The converted sizes are intermediate reduced sizes Mw and Mh. Mh and Mw can be obtained from the relational expression of the following expression (5).

  That is, the head intermediate size calculating means 88 can calculate one set of intermediate reduced sizes Mw and Mh by designating one set of RFw and RFh. However, the size of the front face of humans varies from adults to children, women and men. Therefore, it is possible to prepare a plurality of sets of RFw and RFh and calculate an intermediate reduced size corresponding to each set. By calculating a plurality in advance, the subsequent front face search process can be processed with a plurality of intermediate reduction sizes. Further, it can be interpreted that the search processing with a plurality of intermediate reduction sizes is the same as the act of searching for the head rectangle with a plurality of resolutions.

  When the intermediate size is calculated by the head intermediate size calculation unit 88, the intermediate size information is stored in the intermediate size storage unit 91.

  Next, the minimum inter-pattern distance value Dmin is initialized to a value sufficiently larger than the normally obtained inter-pattern distance value (step S102 in FIG. 22).

  One piece of information in the intermediate reduction size storage unit 91 is selected, and the image reduction unit 90 reduces the head rectangular image to the selected intermediate reduction size to obtain a reduced head rectangular image (step S103 in FIG. 22).

  Next, the front face search positions SX and SY are initialized to 0 (step S104 in FIG. 22).

  Next, the front face candidate extracting means 72 extracts front face candidate images at the search positions SX and SY (step S105 in FIG. 22).

  Next, in order to correct the tone of the front face candidate image, the contrast is corrected by the contrast correcting means 74 (step S106 in FIG. 22).

  An example of a specific method for contrast correction will be described. Assuming that the front face candidate image takes a value from 0 to vmax, the average of the pixel values is μ, and the standard deviation is σ, the conversion formula from the original image V to the contrast corrected image V ′ is the following formula (6 ).

  Referring to FIGS. 10 and 22 again, next, the eigenvector projection distance calculation means 75 obtains the eigenvector projection distance D between the front face candidate image and the standard face pattern (step S107).

  Next, the projection distance minimum judging means 78 compares D and Dmin. At this time, if D is a value smaller than Dmin, D is substituted for Dmin, the value is updated, and stored in the projection distance minimum value storage means 86. At the same time, the front face candidate image is stored in the front face gray value storage means 87 (step S108).

  Next, the search range end determination means 79 increments the search positions SX and SY (step S109), and determines whether or not all the head rectangles have been searched (step S110). If the search has not been completed yet, the process is repeated from step 105 again.

  When the search for the entire search range of the head rectangle has been completed, the multi-resolution processing end determination unit 92 determines whether or not the search has been performed with all intermediate reduced sizes (step S111). If there is an intermediate reduction size that has not been searched, the process starts again from step S103 using a different intermediate reduction size. The front face searching means 71 ends when the search is completed for all intermediate reduced sizes.

  Next, the operation of the eigenvector projection distance calculation means 75 in FIG. 10 will be described.

  The standard face dictionary storage means 76 stores standard face average data and standard face eigenvector data.

  FIG. 33 shows an example of an eigenvector projection distance calculation dictionary when the number of feature quantities is p. The eigenvector projection distance calculation dictionary includes p-dimensional eigenvector data E from the 1st to the p-th and an average value Ave of p feature amounts. When there are p feature quantities, eigenvectors exist up to the p-th, but 1 to m-th are used when calculating the projection distance.

  The pixel values of the front face candidate image are raster scanned as shown in FIG. 8 and converted into one-dimensional feature data. At this time, the product Fw × Fh of the vertical and horizontal sizes of the front face image must be the same as the feature quantity of the dictionary. This is expressed as a vector X: X1, X2,. . . Let Xp.

  First, the average difference means 82 subtracts the average vector Ave from the vector X. This is a vector Y.

  Next, in the vector projection value calculation means 83, the vector Y is projected onto m eigenvectors, and the projection values R1. . Rm is obtained. The projection value calculation method is shown in the following equation (8).

  Next, in the reconstruction calculation means 84 (see FIG. 10), the projection values R1. . . The original feature quantity Y is reconstructed using Rm and m eigenvectors, and the reconstructed vector is Y ′. The reconstruction calculation is shown in the following equation (9).

  Next, the projection distance calculation means 85 calculates the Euclidean distance value between Y and Y ′ according to the following equation (10). Thereby, the projection distance D to the eigenvector E is calculated.

  FIG. 11 is a diagram showing the configuration of the front face searching means 101 as another embodiment of the front face searching means 63 of FIG. Referring to FIG. 11, this front face search means 101 includes a head rectangular image storage means 89, a head intermediate size calculation means 88, an image reduction means 90, an intermediate size storage means 91, and a front face candidate extraction means. 72, intermediate reduced image storage means 73, contrast correction means 74, product-sum operation means 102, standard face dictionary data storage means 104, storage means 105, maximum similarity determination means 103, search range end determination Means 79 and multi-resolution processing end judging means 92 are provided.

  The storage unit 105 includes a similarity maximum value storage unit 106 and a front face gray value storage unit 107.

  FIG. 23 is a flowchart for explaining the processing of the front face search means 101. The operation of the front face searching unit 101 will be described with reference to FIGS. The head rectangular image data is held in the head rectangular image storage unit 89. First, the head intermediate size calculation means 88 calculates the intermediate reduced size of the head rectangular image with reference to the face-to-face distance value and the standard face dictionary data size, and stores it in the intermediate size storage means 91 (step S121).

  The calculation method of the intermediate reduction size is the same as that of the front face search means 71. Next, the maximum similarity Smax is initialized to 0 (step S122).

  One piece of information in the intermediate size storage unit 91 is selected, and the image reduction unit 90 reduces the head rectangular image to the selected intermediate reduction size to obtain a reduced head rectangular image (step S123).

  Next, the front face search positions SX and SY are initialized to 0 (step S124).

  Next, the front face candidate extracting means 72 extracts front face candidate images at the search positions SX and SY (step S125).

  Next, in order to correct the tone of the front face candidate image, the contrast is corrected by the contrast correcting means 74 (step S126).

  Next, the product-sum operation unit 102 obtains the similarity S between the front face candidate image and the standard face pattern (step S127).

  Next, the similarity maximum value determination means 103 compares S and Smax. At this time, if S is a value larger than Smax, S is substituted into Smax, the value is updated, and stored in the similarity maximum value storage unit 106. At the same time, the front face candidate image is stored in the front face gray value storage means 107 (step S128).

  Next, the search range end determination means 79 increments the search positions SX and SY (step S129), and determines whether or not all the head rectangles have been searched (step S130). If the search has not been completed yet, the process is repeated from step 125 again.

  When the search for the entire search range of the head rectangle has been completed, the multi-resolution processing end determination unit 92 determines whether or not the search has been performed with all intermediate reduction sizes. If there is an intermediate reduction size that has not been searched, the process starts again from step S123 using a different intermediate reduction size.

  When the search is completed for all intermediate reduction sizes, the front face search means 101 ends.

  Next, the operation of the product-sum operation unit 102 for calculating the similarity with the standard face pattern shown in FIG. 11 will be described.

  The product-sum operation unit 102 calculates the similarity S with reference to a linear discrimination dictionary that discriminates whether the face is the front face or not. FIG. 34 shows an example of the linear discrimination dictionary. FIG. 34 shows a dictionary that discriminates q classes, but the standard face dictionary data storage means 104 is a dictionary that discriminates the front face and the other two classes when q = 2. It corresponds to.

  The pixel values of the front face candidate image are raster scanned as shown in FIG. 8 and converted into one-dimensional feature data. At this time, the product Fw × Fh of the vertical and horizontal sizes of the front face image must be the same as the feature quantity of the dictionary. This is expressed as a vector X: X1, X2,. . . Let Xp.

  Also, as standard face dictionary data, a class with q = 1 represents a front face, and a class with q = 2 represents the other. The similarity with the front face can be calculated by the following equation (11) using only the q = 1 row of FIG. 34, that is, the class 1 identification coefficient 422.

  The product-sum operation unit 102 calculates the similarity by calculating the above equation (11).

  13 includes a face identification unit 111 that is an example of the face identification unit 9 of FIG. 1, an identification dictionary storage unit 112 that is an example of the identification dictionary storage unit 10 of FIG. 1, and an identification result of FIG. An identification result correction unit 113 which is an embodiment of the correction unit 11 is shown. Referring to FIG. 13, the face identification unit 111 includes a feature data storage unit 115, a product-sum operation unit 116, a maximum similarity person determination unit 118, and a threshold processing unit 117. The identification dictionary storage unit 112 includes a registered person identification dictionary storage unit 119. The identification result correction unit 113 includes an identification result weighted average calculation unit 114.

  FIG. 24 is a flowchart for explaining the processing of the face identifying unit 111 and the identification result correcting unit 113. With reference to FIGS. 13 and 24, operations of the face identifying unit 111 and the identification result correcting unit 113 will be described.

  Feature data is input and stored in the feature data storage means 115 (step S51).

  Next, referring to the data in the registered person identification dictionary storage means 119, the product-sum operation means 116 calculates the similarity to the registered person for each person (step S54).

  The method of calculating the similarity is basically the same as the operation of the product-sum operation unit 102 that calculates the similarity with the standard face pattern. However, the number of classes to be identified is the number of registered persons.

  When data for q people is registered as a dictionary, the registered person identification dictionary storage means 119 holds the same number of data as the linear discrimination dictionary shown in FIG. Then, the product-sum operation means 116 obtains q similarity degrees Si: (i = 1,... Q) as shown in the following equation (12).

  A method of identifying a pattern based on the magnitude of similarity obtained by the product-sum operation processing using the linear discrimination dictionary shown in FIG. 34 is referred to as “similarity discrimination using the linear discrimination dictionary”.

  Referring to FIGS. 13 and 24 again, next, the maximum similarity person determination unit 118 obtains the maximum value among the calculated q similarity degrees, and obtains the person corresponding to the maximum value (step S55). That is, the person who is judged to be most similar to the feature data is obtained. The similarity at this time is called “maximum similarity”.

  Next, the threshold processing means 117 compares the maximum similarity with a predetermined threshold (step S56).

  When the maximum similarity is higher than the threshold value, the face identification unit 111 determines that the person has been identified, and outputs the ID number and the maximum similarity. When the maximum similarity is lower than the threshold value, there is a high possibility that the person is not a registered person (person), so information “other person” is output.

  After receiving the identification result, the identification result correcting unit 113 integrates the identification results in the past N frames by the identification result weighted average calculating unit 114 (step S57). As an example of the operation of the identification result weighted average calculating means 114, there is a method of performing the weighted average of the identification person ID and similarity in the past N frames or the other person determination result as follows.

  Step A1: If the number of frames at a certain ratio in the past N frames is “other”, it is determined as “other”. If it is determined that it is not another person, the process proceeds to step A2.

  Step A2: It is assumed that there is a Ni frame in the past N frames that is determined as a person i (i = 1 ... q). The similarity weighted average value of each person is calculated by the following equation (13). Si represents the similarity of person i, and SSi represents the similarity weighted average value of person i. The person ID with the largest SSi is output as the identification result.

  The identification result correcting unit 113 outputs the identification result integrated as described above.

FIG. 15 shows the configuration of the face identifying means 131 which constitutes another embodiment of the face identifying means 9 of FIG.
It is a figure which shows the structure of the identification dictionary memory | storage means 132 which makes another Example of the identification dictionary memory | storage means 10 of FIG. Referring to FIG. 15, the face identification unit 131 includes a feature data storage unit 115, an eigenvector / other person determination unit 133, a product-sum operation unit 116, a maximum similarity person determination unit 118, and a threshold processing unit 117. I have.

  The identification dictionary storage unit 132 includes a different person determination dictionary storage unit 134 and a registered person identification dictionary storage unit 119.

  FIG. 24 is a flowchart for explaining the processing of the face identification unit 131. The operation of the face identification unit 131 will be described with reference to FIGS. Feature data is input and stored in the feature data storage means 115 (step S51).

  Next, the inter-pattern distance Dh with the registered person group is obtained by the eigenvector / other person discriminating means 133 while referring to the data in the other person discrimination dictionary storage means 134 (step S52), and the inter-pattern distance Dh is the threshold. If it is larger than the value, it is determined that the person is another person (step S53). The inter-pattern distance Dh is calculated as an eigenvector projection distance. That is, the eigenvector dictionary created based on the feature data of all registered persons is stored in the other person discrimination dictionary storage means.

  An example of the eigenvector dictionary is shown in FIG. 33, and the eigenvector projection distance calculation method is described in the explanation of the operation of the eigenvector projection distance calculation means 75.

  If the input feature data is determined to be another person by the eigenvector / other person determination means 133, the face identification means 131 immediately outputs “other person” without going through the product-sum operation means 116.

  If the eigenvector other person determination means 133 does not determine “others”, the data in the registered person identification dictionary storage means 119 is referred to, and the product sum calculation means 116 determines the similarity to the registered person. Are calculated for each person (step S54). If q persons are registered, q similarity degrees Si: (i = 1,... q) are calculated.

  Next, the maximum similarity person determination unit 118 obtains the maximum value among the calculated q similarity degrees, and obtains a person corresponding to the maximum value (step S55). That is, the person who is judged to be most similar to the feature data is obtained. The similarity at this time is called the maximum similarity. Next, the threshold processing means 117 compares the maximum similarity with a predetermined threshold (step S56).

  When the maximum similarity is higher than the threshold value, the face identification unit 131 determines that the person has been identified, and outputs the ID number and the maximum similarity. When the maximum similarity is lower than the threshold value, there is a high possibility that the person is not a registered person, so the information “other person” is output.

  FIG. 14 is a diagram illustrating a configuration of the dictionary data management unit 121 which is an embodiment of the dictionary data management unit 12 of FIG. Referring to FIG. 14, the dictionary data management unit 121 includes a personal feature data storage unit 122, an identification dictionary generation unit 123, and a selector 124.

  The identification dictionary generation unit 123 includes a linear discrimination dictionary creation unit 126 and an eigenvector dictionary creation unit 127. In the personal feature data storage unit 122, there are personal feature data regions 125 for the number of registered persons.

  When the feature data and the person ID number are input, the dictionary data management unit 121 assigns the input feature data to an area corresponding to the person ID number in the personal feature data storage unit 122 by the selector 124. Remember. If there is an instruction to add a new person ID, a new person-specific feature data area 125 is secured in the personal feature data storage unit 122 and a new person ID number is assigned. If there is an instruction to delete an existing person ID, the person-specific feature data area 125 of the corresponding ID in the personal feature data storage unit 122 is discarded.

  When the dictionary data management unit 121 receives the identification dictionary creation command, the identification dictionary generation unit 123 uses the data of the personal feature data storage unit 122 to create a registered person identification identification dictionary and an eigenvector dictionary, which are linear discrimination dictionaries. A certain person discrimination dictionary is generated. The registered person identifying identification dictionary is created by the linear discrimination dictionary creating means 126. The other person discrimination dictionary is created by the eigenvector dictionary creating means 127.

  When the face identification unit 9 in FIG. 1 has the configuration of the face identification unit 111 shown in FIG. 13, the other person discrimination dictionary is unnecessary, and therefore the identification dictionary generation unit 123 has the eigenvector dictionary creation unit 127. There may be no configuration.

  FIG. 25 is a flowchart for explaining the registration process in the dictionary data management unit 12 of FIG. The operation when the dictionary data management unit 121 registers a new person in front of the camera will be described with reference to FIGS.

  The person detection / identification manager 13 designates a new person ID and instructs the dictionary manager 121 to register a new person (step S61).

  The dictionary data management unit 121 secures a personal feature data area 125 corresponding to the specified ID in the personal feature data storage unit 122 (step S62).

  The person detection / identification management unit 13 acquires the specified number of feature data from the face feature extraction unit 8 in the person detection / identification unit 1 and sends it to the dictionary data management unit 121.

  The dictionary data management unit 121 stores the acquired data in the feature data area 125 for each person corresponding to the new ID (step S63).

  When the acquisition for the designated number is completed, the person detection / identification management unit 13 instructs the dictionary data management unit 121 to create an identification dictionary (step S64).

  Upon receiving the creation instruction, the dictionary data management unit 121 creates a registered person identification dictionary by the linear discrimination dictionary creation unit 126 of the identification dictionary creation unit 123 (step S65).

  Next, another person discrimination dictionary is created by the eigenvector dictionary creating means 127 (step S66).

  Then, the created dictionary is output and stored in the identification dictionary storage means 10 of the person detection / identification unit (step S67). With the above processing, the new person registration processing is completed.

  FIG. 29 is a diagram showing the configuration of the linear discrimination dictionary creation means 311 that constitutes an embodiment of the linear discrimination dictionary creation means 126 of FIG. Referring to FIG. 29, the linear discriminant dictionary creation means 311 includes a feature amount X storage means 312, a variance covariance matrix Cxx calculation means 313, an inverse matrix conversion means 314, a matrix multiplication means 315, and an objective variable Y storage means. 317, objective variable Y generation means 318, covariance matrix Cxy calculation means 319, coefficient storage means 320, and constant term calculation means 316.

  A method for creating a linear discrimination dictionary will be described with reference to FIGS. 31 and 34. FIG. 31 shows n individual feature data X341 for n persons, from person 1 to person q. The number of features of the personal feature data X341 is p. One row in FIG. 31 shows the feature data for one sheet. The personal feature data X341 is stored in the personal feature data storage unit 122 in FIG. 14, and the personal feature data for one person is stored in the personal feature data area 125, respectively.

  The objective variable Y342 is a vector that is created for each feature data and has elements for the number of persons to be identified. That is, in FIG. 31, since the person ID takes a value from 1 to q, the objective variable Y exists from Y1 to Yq. The value of the objective variable Y342 is a binary value of 0 or 1, the vector element of the person to which the feature data belongs is 1, and the others are 0. That is, in the case of the feature data of the person 2, only the Y2 element is 1 and the others are 0.

  FIG. 34 is a diagram illustrating an example of the format of the linear discrimination dictionary. The linear discrimination dictionary is composed of two types of coefficients, a constant term 421 and a multiplication term 425. If the matrix consisting of multiplication terms is Aij (i = 1,... P, j = 1,... Q) and the vector consisting of constant terms is A0j (j = 1,... Q), then the matrix. Aij is obtained from the following equation (14).

  In the above equation (14), Cxx is this variance-covariance matrix using all data of the personal feature data X341. This variance covariance matrix Cxx is calculated by the following equation (15).

  Elements of the personal feature data X are represented by xij: (i = 1... N, j = 1,... P). N is the total number of data and p is the number of features. In the example shown in FIG. 31, since there are n pieces of data per person, N = nq. x￣ represents the average value of x.

  Cxy is a covariance matrix of the individual feature data X and the objective variable Y. The covariance matrix Cxy is calculated according to the following equation (16). x￣ and y￣ represent average values of x and y.

  The constant term A0j is calculated according to the following equation (17).

  First, Cxx is calculated and its inverse matrix is obtained. Next, Cxy is obtained. Finally, these matrixes are multiplied to obtain a multiplication term matrix Aij, and finally a constant term A0j is obtained.

  FIG. 29 is a diagram for explaining the processing of the linear discrimination dictionary creation means 311. With reference to FIG. 29, the operation of the linear discrimination dictionary creation means 311 will be described. The input personal feature data group is stored in the feature amount X storage unit 312.

  The objective variable Y is generated by the objective variable Y generation means 318 using the inputted personal feature data group and person ID. The generated objective variable Y is stored in the objective variable Y storage means 317.

  Next, the variance-covariance matrix Cxx calculation means 313 calculates the variance-covariance matrix Cxx.

  Next, an inverse matrix conversion unit 314 calculates an inverse matrix of the variance-covariance matrix Cxx.

  Next, the covariance matrix Cxy calculation means 319 calculates the covariance matrix Cxy.

  Next, the matrix multiplication unit 315 calculates the multiplication term Aij, and stores the multiplication term Aij data in the coefficient storage unit 320.

  Next, the constant term calculation unit 316 calculates the constant term A0j and stores it in the coefficient storage unit 320.

  Finally, the data of the coefficient storage means 320 is output and the process ends.

  In FIG. 31, binary data of 0 and 1 is shown. However, in the linear discriminating dictionary creation means 311, other binary data such as 0 and 100 can be used.

  FIG. 30 is a diagram showing an eigenvector dictionary creating unit 331 that constitutes an embodiment of the eigenvector dictionary creating unit 127 of FIG. The eigenvector dictionary creation unit 331 includes a feature amount storage unit 332, a feature amount average calculation unit 333, a variance covariance matrix calculation unit 334, an eigenvector calculation unit 335, and a coefficient storage unit 336.

  A method for creating an eigenvector dictionary will be described with reference to FIGS. 31 and 33. FIG. Elements of the personal feature data X are represented by xij: (i = 1... N, j = 1,... P).

  First, an average value for each feature amount of X is obtained. Next, the variance covariance matrix Cxx of X is calculated using the above equation (15). The eigenvector of the variance-covariance matrix Cxx is obtained. Methods for obtaining eigenvectors are widely known by those skilled in the art and are not directly related to the present invention, and therefore, details thereof are omitted.

  Through the above operation, an eigenvector dictionary having a format as shown in FIG. 33 is obtained. Eigenvectors are obtained by the number of feature quantities (p). In FIG. 33, one row represents one eigenvector.

  FIG. 30 is a diagram for explaining the configuration and processing of the eigenvector dictionary creating unit 331. The operation of the eigenvector dictionary creation unit 331 will be described with reference to FIG.

  When personal feature data is input, it is stored in the feature amount storage means 332.

  Next, in the feature quantity average calculation means 333, an average value of the feature quantities is obtained and stored in the coefficient storage means 336. Next, the variance / covariance matrix calculation means 334 calculates the variance / covariance matrix Cxx.

  Next, the eigenvector calculation means 335 calculates the eigenvector of the variance-covariance matrix Cxx and stores it in the coefficient storage means 336. Finally, the data of the coefficient storage means 336 is output and the process ends.

  FIG. 16 is a diagram showing a configuration of an embodiment of a robot apparatus according to the present invention. Referring to FIG. 16, the robot apparatus 201 includes a CCD camera 202, a person detection / identification unit 203, a dictionary data management unit 204, a person detection / identification management unit 205, an overall control unit 206, a speaker 207, and a robot. Moving means 208. The robot moving unit 208 includes a motor 209 and wheels 210.

  The person detection / identification means 203 performs person detection and identification based on the stereo video from the CCD camera 202. The person detection / identification management unit 205 exchanges information with the person detection / identification means 203, exchanges information with the overall control unit 206, and exchanges information with the dictionary data management unit 204. The speaker 207 is connected to the overall control unit 206 and can speak in response to an instruction from the overall control unit 206. The overall control unit 206 also sends a movement instruction to the robot moving means 208. The robot moving means 208 has a motor 209 and a plurality of wheels 210, and can move the robot in any direction.

  FIG. 26 is a diagram for explaining the process of the embodiment of the robot apparatus according to the present invention. With reference to FIG.16 and FIG.26, operation | movement of the robot apparatus 201 of one embodiment of this invention is demonstrated. When the robot device 201 detects a person, it moves in the direction of the person and approaches the person, and performs person identification when approaching within a predetermined distance.

  The person detection / identification management unit 205 acquires the detected rectangle information of the head and the facing distance value from the head detection / tracking means in the person detection / identification means 203, and transmits them to the overall control section 206 (step S71).

  The overall control unit 206 refers to the facing distance value and determines whether the facing distance is closer than a threshold value (step S73). When the distance is longer than the threshold value, the robot moving unit 208 is instructed to move forward in the direction of the person (step S72).

  The approximate direction in which a person exists can be inferred from the head rectangular coordinates in the image. Steps S71 to S73 are repeated, and when the facing distance becomes closer than the threshold value, the overall control unit 206 acquires a person identification result from the person detection / identification management unit 205 (step S74).

  Next, a voice for each person is uttered from the speaker 207 to inform the conversation person that the person has been identified (step S75).

  FIG. 17 is a diagram showing the configuration of an embodiment of the robot apparatus of the present invention. Referring to FIG. 17, the robot apparatus 221 includes a CCD camera 202, a face-to-face distance sensor 223, a touch sensor 222, a person detection / identification unit 224, a dictionary data management unit 204, a person detection / identification management unit 205, A control unit 227, a microphone 225, a voice recognition unit 226, a speaker 207, and a robot moving unit 208 are provided. The robot moving unit 208 includes a motor 209 and wheels 210.

  The person detection / identification means 224 performs person detection and identification based on information from the stereo image from the CCD camera 202 and the face-to-face distance sensor 223. The person detection / identification management unit 205 exchanges information with the person detection / identification means 224, exchanges information with the overall control unit 227, and exchanges information with the dictionary data management unit 204. The speaker 207 is connected to the overall control unit 227 and can speak in response to an instruction from the overall control unit 227. Further, the overall control unit 227 sends a movement instruction to the robot moving means 208. The robot moving means 208 has a motor 209 and a plurality of wheels 210, and can move the robot in any direction. The touch sensor 222 is connected to the overall control unit 227 and detects the presence / absence of an object contact and the strength of the contact from the outside. The microphone 225 is connected to the voice recognition unit 226, and the voice recognition unit 226 is connected to the overall control unit 227. The voice recognition unit 226 automatically recognizes a person's word from the voice data from the microphone 225 and transmits the recognition result to the overall control unit.

  FIG. 27 is a flowchart for explaining processing of the robot apparatus according to the embodiment of the present invention. An operation example of the robot apparatus 221 according to the embodiment of the present invention will be described with reference to FIGS.

  The robot device 221 detects and identifies the person who is facing, and updates the identification dictionary according to the reaction of the interlocutor to sequentially learn the person images.

  The overall control unit 227 acquires the person identification result from the person detection / identification management unit 205 (step S81).

  Next, a specific operation is performed for each identified person (step S82). A specific action refers to the entire action of speaking a person's name or moving a wheel in a specific direction by a person.

  Next, it waits for the interaction of the dialogue person to be sensed (step S83). When the interaction is obtained, it is determined whether the response is positive or negative (step S84). The positive reaction means “Yes”, for example, when “Yes” is recognized by voice recognition.

  The negative reaction means “No”, for example, when “No” is recognized by voice recognition.

  Also, for example, by pre-determining the rule “Yes” when the touch sensor is pressed once in advance and “No” when it is pressed twice in advance, the response of the conversation person is acquired using the touch sensor. Can do.

  If step S84 is negative, the operation is repeated once again from step S81. In the case of a positive reaction, the facial feature data used for identification is input to the dictionary data management unit 204 (step S85).

  Then, an identification dictionary is created and updated (step S86).

  If there is an end instruction, the process ends. If not, the operation is repeated from step S81 (step S87).

  FIG. 28 is a flowchart for explaining the processing of another embodiment of the robot apparatus of the present invention. An example of the operation of the robot apparatus 221 according to another embodiment of the present invention will be described with reference to FIGS. The robot device 221 detects and identifies the person who is facing the person, acquires an image of a new person according to the command of the conversation person, and registers a new person dictionary.

  The overall control unit 227 acquires a person identification result from the person detection / identification management unit 205 (step S91).

  Next, a specific operation is performed for each identified person (step S92). A specific action refers to the entire action of speaking a person's name or moving a wheel in a specific direction by a person.

  Next, it waits for the command of the interactor to be sensed (step S93). When a dialogue event is obtained, it is next determined whether or not the command event is a registration command (step S94). The registration command event includes, for example, an event that “Toroku” is recognized by voice recognition. In addition, a rule that a registration event is determined by tapping the touch sensor once in advance is determined in advance by the overall control unit, so that a registration command event can be uttered using the touch sensor.

  When a registration command is received, registration processing is performed, and a new person is registered (step S95). An example of the registration processing step S95 is shown in FIG. 25 and has already been described.

  When the registration process ends, the process starts again from step S91. If the registration command event does not come, an end determination is made (step S96).

  If there is an end instruction, the process ends. If there is no end instruction, the process starts again from step S91.

  FIG. 35 is a diagram showing a configuration of an embodiment of a person detection and identification system according to the related invention of the present invention. Referring to FIG. 35, the person detection and identification system according to the related invention of the present invention includes a video acquisition unit 2, a face-to-face distance sensor 5, a person detection and identification unit 501, a dictionary data management unit 12, and a person detection and identification management unit. 502 and a recording medium 503. The recording medium 503 records a person detection / identification processing program, and may be a magnetic disk, a semiconductor memory, a CD-ROM, or another recording medium.

  The person detection / identification processing program is read from the recording medium 503 into the person detection / identification unit 501 and the person detection / identification management unit 502, and the person detection / identification unit 6 and the person detection in the embodiment described above with reference to FIG. The same processing as that performed by the identification management unit 13 is executed.

It is a block diagram which shows the structure of one Embodiment of the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the head detection tracking means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of another Example of the head detection tracking means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the monocular head rectangular coordinate detection means in the person identification apparatus used with the robot apparatus of this invention. It is a figure for demonstrating the head detection process in the person identification apparatus used with the robot apparatus of this invention. It is a figure for demonstrating the right-and-left image collation process in the person identification device used with the robot apparatus of this invention. It is a figure for demonstrating the head tracking process in the person identification apparatus used with the robot apparatus of this invention. It is explanatory drawing of the raster scan at the time of converting the front face image into the feature data in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the front face alignment means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the front face search means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of the other Example of the front face search means in the person identification device used with the robot apparatus of this invention. It is a figure for demonstrating the head detection process and face position alignment process in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of each Example of the face identification means, identification dictionary memory | storage means, and identification result correction | amendment means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the dictionary data management part in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of each Example of the face identification means, identification dictionary memory | storage means, and identification result correction | amendment means in the person identification device used with the robot apparatus of this invention. It is a figure which shows the structure of one Example of the robot apparatus of this invention. It is a figure which shows the structure of the other Example of the robot apparatus of this invention. It is a flowchart for demonstrating the person detection process and identification process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the head detection tracking process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the head detection process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the front face alignment process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the front face search process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the front face search process in the person identification device used with the robot apparatus of this invention. It is a flowchart for demonstrating the face identification process in the person identification apparatus used with the robot apparatus of this invention. It is a flowchart for demonstrating the person dictionary registration process in the person identification apparatus used with the robot apparatus of this invention. It is a flowchart for demonstrating action control in the robot apparatus of this invention. It is a flowchart for demonstrating action control in the robot apparatus of this invention. It is a flowchart for demonstrating action control in the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the linear discrimination dictionary creation means in the person identification device used with the robot apparatus of this invention. It is a block diagram which shows the structure of one Example of the eigenvector dictionary preparation means in the person identification device used with the robot apparatus of this invention. It is a figure for demonstrating the linear discrimination dictionary creation method used for the person identification process used with the robot apparatus of this invention. It is a figure for demonstrating the facing distance calculation method by the stereo vision used for the person identification process used with the robot apparatus of this invention. It is explanatory drawing of the eigenvector projection distance dictionary used for the person identification process used with the robot apparatus of this invention. It is explanatory drawing of the linear discrimination dictionary used for the person identification process used with the robot apparatus of this invention. It is a block diagram which shows the structure of the other Example of the person identification apparatus used with the robot apparatus of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Person detection identification means 2 Image | video acquisition means 3 Right camera 4 Left camera 5 Face-to-face distance sensor 6 Head detection tracking means 7 Front face alignment means 8 Face feature-value extraction means 9 Face identification means 10 Identification dictionary memory means 11 Identification result correction | amendment Means 12 Dictionary data management unit 13 Human detection identification management unit 14 Human identification device 21 Head detection unit 22 Head tracking unit 23 Head rectangle storage unit 24 Monocular head rectangular coordinate detection unit 25 Left and right image collation unit 26 Face-to-face distance evaluation Means 27 Head detection tracking means 28 Right camera 29 Left camera 30 Face-to-face distance sensor 31 Face-to-face distance integration means 32 Head detection tracking means 33 Head detection means 34 CCD camera 41 Head rectangular coordinate detection means 42 Motion pixel detection means 43 Person Number evaluation means 44 Head vertex coordinate detection means 45 Head lower coordinate detection means 46 Temporal coordinate detection means 4 Motion region width 48 integrated difference image G
49 Integrated difference image G
50 Integrated Difference Image Upper Area 51 Head Detection Rectangle 52 Search Result 53 Head Rectangle Coordinate of Previous Frame 54 Search Trajectory 55 Head Rectangle Image of Front Frame 61 Front Face Positioning Unit 62 Head Rectangle Cutting Unit 63 Front Face Search Unit 64 Standard face dictionary data storage means 65 Front face appearance determination means 66 Density dispersion determination means 67 Threshold processing means 71 Front face search means 72 Front face candidate extraction means 73 Intermediate reduced image storage means 74 Contrast correction means 75 Eigenvector projection Distance calculation means 76 Standard face dictionary storage means 77 Storage means 78 Projection distance minimum determination means 79 Search range end determination means 80 Standard face average data storage means 81 Standard face eigenvector data storage means 82 Average difference means 83 Vector projection value calculation means 84 Configuration calculation means 85 Projection distance calculation means 8 Projection distance minimum value storage means 87 Front face gray value storage means 88 Head intermediate size calculation means 89 Head rectangular image storage means 90 Image reduction means 91 Intermediate size storage means 92 Multi-resolution processing end determination means 101 Front face search means 102 Product Sum calculation means 103 Similarity maximum determination means 104 Standard face dictionary data storage section 105 Storage means 106 Similarity maximum value storage means 107 Front face gray value storage means 111 Face identification means 112 Identification dictionary storage means 113 Identification result correction means 114 Identification result Weighted average calculation means 115 Feature data storage means 116 Product-sum operation means 117 Threshold processing means 118 Maximum similarity person determination means 119 Registered person identification dictionary storage means 121 Dictionary data management section 122 Personal feature data management section 123 Identification dictionary generation Means 124 Selector 125 Person Feature data 126 Linear discrimination dictionary creation means 127 Eigenvector dictionary creation means 131 Face identification means 132 Identification dictionary storage means 133 Other person discrimination means 134 Other person discrimination dictionary storage means 151 Head rectangle 152 Head rectangle image 153 Reduced head rectangle image 154 Front Face search shape 155 Front face image 201 Robot apparatus 202 CCD camera 203 Human detection identification means 204 Dictionary data management section 205 Human detection identification management section 206 Overall control section 207 Speaker 208 Robot movement means 209 Motor 210 Wheel 221 Robot apparatus 222 Touch sensor 223 Face-to-face distance sensor 224 Person detection / identification means 225 Microphone 226 Voice recognition means 227 Overall control unit 311 Linear discriminant dictionary creation means 312 Feature amount X storage means 313 Variance covariance matrix Cxx calculation means 314 Inverse matrix conversion means 315 Matrix multiplication means 316 Constant term calculation means 317 Objective variable Y storage means 318 Objective variable Y generation means 319 Covariance matrix Cxy calculation means 320 Coefficient storage means 331 Eigenvector dictionary creation means 332 Feature quantity storage means 333 Feature quantity Mean calculation means 334 Variance covariance matrix calculation means 335 Eigenvector calculation means 336 Coefficient storage means 341 Individual feature data X
342 Objective variable Y
343 Feature data average value 344 Objective variable average value 401 Right camera 402 Left camera 403 Object 411 First eigenvector 412 Second eigenvector 413 Pth eigenvector 414 Average value 421 Constant term 422 Class 1 identification coefficient 423 Class 2 Identification coefficient 424 Class q identification coefficient 425 Multiply term 501 Person detection identification processing section 502 Person detection identification management section 503 Recording medium

Claims (3)

Head detection and tracking means for detecting a human head from the acquired image information;
Means for obtaining a distance from the detected head to at least one of the front objects;
Face identifying means for identifying a person using the detected image of the head;
Means for moving;
A robot device with a face identification function comprising:
When the distance is greater than or equal to a threshold value, a robot apparatus comprising means for moving so as to approach the object for which the distance has been acquired.   2. The robot according to claim 1, wherein the means for acquiring the distance further acquires the direction of the detected head and controls the moving direction in the means for moving so as to approach the object for which the distance has been acquired. apparatus. Head detection and tracking means for detecting a human head from the acquired image information;
Means for obtaining a distance from the detected head to at least one of the front objects;
Face identifying means for identifying a person using the detected image of the head;
Means for moving;
A robot device with a face identification function comprising:
A robot apparatus comprising: means for moving away from the object from which the distance is acquired when the distance is less than or equal to a threshold value.

Images