JP2011221602A - Face search device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operation performance.SOLUTION: A CPU 26 sequentially designates one or more face images detected by searching the face images from search image data stored in a search image area 32c of an SDRAM 32 thereby to sequentially designate the found face image. The CPU 26 detects an expression of a designated face image to search a registered face image that coincides with the designated face image among a plurality of registered face images with reference to the detected expression. The CPU 26, after that, displays attribute information showing an attribute of the detected registered face image on an LCD monitor 38.

Description

  The present invention relates to a face search device, and more particularly to a face search device that searches a registered face image that matches a face image appearing in an object scene image from a plurality of registered face images.

  An example of this type of device is disclosed in Patent Document 1. According to this background art, the face that appears in the scene is detected based on the image data output from the image sensor that captures the scene. The detected face is subjected to identification processing as to whether or not it is common with a smile set in advance, and is determined as a smile determination target when the identification result is affirmative. The image data output from the image sensor is recorded on the storage medium when the smile determination target face changes to a smile.

JP 2009-177678 A

  However, in the background art, the facial expression detected from the object scene is not referred to when executing the identification processing whether or not it is common with a preset smile. For this reason, depending on the facial expression detected from the object scene, the operation performance may be degraded.

  Therefore, a main object of the present invention is to provide a face search device capable of improving operation performance.

  A face search device according to the present invention (10: reference numerals corresponding to the embodiments; the same applies hereinafter) includes first search means (S153 to S183) for searching for one or more face images from an object scene image, the first search First specifying means (S189) for designating each of the one or more face images found by the means as a first noticeable face image, and a registered face image matching the first noticeable face image as an expression of the first noticeable face image 2nd search means (S201 to S207, S217 to S227, S231 to S249, S251 to S263, S273 to S279) for searching from a plurality of registered face images with reference to the above, and registration found by the second search means Output means (S209 to S215, S265 to S271, S125 to S133) for outputting attribute information indicating the attribute of the face image is provided.

  Preferably, the second search means includes second designation means (S205, S227) for designating each of the plurality of registered face images as the second attention face image, and the expression of the first attention face image and the second attention face image. Correction means (S219) for correcting the first attention face image and / or the second attention face image so as to suppress the difference from the facial expression is included.

  More preferably, each of the plurality of registered face images is accompanied by a plurality of expression parameters, and the second search means selects an expression parameter of the second attention face image from among the plurality of expression parameters for correction processing by the correction means. It further includes parameter selection means (S217).

  Preferably, the second search means designates each of the plurality of registered face images as a second noticeable face image (S205, S249), the first noticeable face image and / or so as to show different facial expressions. Correction means (S231 to S235, S239 to S241) for repeatedly correcting the second attention face image, collation means (S237) for comparing the first attention face image and the second attention face image for each correction processing by the correction means, and Specific means (S243 to S247) for specifying a registered face image that coincides with the one face image of interest based on the collation result of the collation means is included.

  Preferably, each of the plurality of registered face images includes a plurality of images representing a common face with different facial expressions, and the second search means uses the second designation means to select an image having a facial expression along the facial expression of the first attention face image. The image selecting means (S253 to S257) for selecting from among a plurality of images in relation to the designation processing, and the collating means (S273) for collating the image selected by the image selecting means with the first target pixel image.

  Preferably, each of the plurality of registered face images has a plurality of names, and the attribute information output by the output unit includes the name of the registered face image found by the second search unit.

  Preferably, an imaging means (16) for capturing the scene is further provided, and the first search means executes a search process on the scene image output from the imaging means.

  Preferably, the first search means executes search processing with reference to the standard face image.

  A face search program according to the present invention includes a first search step (S153 to S183) and a first search step for searching for one or more face images from an object scene image in a processor (26) of a face search device (10). In step S189, each of the one or more face images discovered by step S is designated as a first attention face image, and the registered face image that matches the first attention face image is referred to the expression of the first attention face image. A second search step (S201 to S207, S217 to S227, S231 to S249, S251 to S263, S273 to S279) for searching from a plurality of registered face images and the registered face images found by the second search step A face search program for executing an output step (S209 to S215, S265 to S271, S125 to S133) for outputting attribute information indicating an attribute.

  The face search method according to the present invention is a face search method executed by the face search device (10), wherein a first search step (S153 to S183) for searching for one or more face images from an object scene image, A designating step (S189) of designating each of the one or more face images found in the first search step as a first target face image, and a registered face image that matches the first target face image as the first target face image. Discovered by the second search step (S201 to S207, S217 to S227, S231 to S249, S251 to S263, S273 to S279) for searching from a plurality of registered face images with reference to facial expressions, and the second search step An output step (S209 to S215, S265 to S271, S125 to S133) for outputting attribute information indicating the attribute of the registered face image is provided.

  According to the present invention, when searching for a registered face image that matches the face image appearing in the scene image, the facial expression of the face image appearing in the scene image is referred to. The output attribute information indicates the attribute of the registered face image discovered in this way, that is, the attribute of the face image that appears in the object scene image. As a result, the operating performance is improved.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of this invention. It is a block diagram which shows the structure of one Example of this invention. It is an illustration figure which shows an example of the registration frame displayed on a monitor screen in face registration mode. It is an illustration figure which shows an example of the allocation state of the evaluation area in an imaging surface. It is an illustration figure which shows an example of a structure of the standard face dictionary applied to the FIG. 2 Example. FIG. 5 is an illustrative view showing one example of a configuration of a face frame register applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of the face detection frame used in the face detection task for registration, and the face detection task for imaging. It is an illustration figure which shows an example of the face detection process in the registration face detection task. (A) is an illustration figure which shows an example of the object scene image displayed on the monitor screen in face registration mode, (B) is another example of the object scene image displayed on the monitor screen in face registration mode. (C) is an illustrative view showing another example of the object scene image displayed on the monitor screen in the face registration mode, and (D) is an object view displayed on the monitor screen in the face registration mode. It is an illustration figure which shows another example of a scene image. It is an illustration figure which shows an example of the registration operation screen displayed on a monitor screen in face registration mode. It is an illustration figure which shows an example of a structure of the registration face dictionary applied to the FIG. 2 Example. It is an illustration figure which shows an example of the face detection process in the face detection task for imaging. FIG. 3 is an illustrative view showing one example of an object scene captured by the embodiment in FIG. 2; (A) is an illustrative view showing an example of a registered face image before correction, and (B) is an illustrative view showing an example of a registered face image after correction. It is an illustration figure which shows an example of a structure of the recognition face register applied to the FIG. 2 Example. FIG. 5 is an illustrative view showing one example of a configuration of a confirmation register applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of the object scene image displayed on the monitor screen in imaging mode. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of a structure of the registration face dictionary applied to another Example. It is a flowchart which shows a part of operation | movement of CPU applied to another Example. It is a flowchart which shows a part of other operation | movement of CPU applied to another Example. It is a flowchart which shows a part of other operation | movement of CPU applied to another Example. It is an illustration figure which shows an example of a structure of the registration face dictionary applied to another Example. It is a flowchart which shows a part of operation | movement of CPU applied to another Example. It is a flowchart which shows a part of other operation | movement of CPU applied to another Example. It is a flowchart which shows a part of other operation | movement of CPU applied to another Example.

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the face search device of the present invention is basically configured as follows. The first search means 1 searches for one or more face images from the object scene image. The first designation unit 2 designates each of one or more face images discovered by the first search unit 1 as a first target face image. The second search means 3 searches the registered face image matching the first attention face image from the plurality of registered face images with reference to the facial expression of the first attention face image. The output unit 4 outputs attribute information indicating the attribute of the registered face image found by the second search unit 3.

When searching for a registered face image that matches the face image that appeared in the object scene image, the facial expression of the face image that appeared in the object scene image is referred to. The output attribute information indicates the attribute of the registered face image discovered in this way, that is, the attribute of the face image that appears in the object scene image. As a result, the operating performance is improved.
[Example]

  Referring to FIG. 2, the digital camera 10 of this embodiment includes a focus lens 12 and an aperture unit 14 driven by drivers 18a and 18b, respectively. The optical image of the object scene that has passed through these members is irradiated onto the imaging surface of the imager 16 and subjected to photoelectric conversion. As a result, a charge representing the object scene image is generated.

  When the power is turned on, the CPU 26 determines the setting (that is, the current operation mode) of the mode changeover switch 28md provided in the key input device 28 under the main task. If the current operation mode is the face registration mode, a face registration task and a registration face detection task are activated. If the current operation mode is the imaging mode, the imaging task and the imaging face detection task are activated.

  When the face registration mode is selected, the CPU 26 gives a registration frame display command to the graphic generator 46 under the face registration task. The graphic generator 46 creates graphic data representing the registration frame RF1 and gives the created graphic data to the LCD driver 36. The registration frame RF1 is displayed on the LCD monitor 38 in the manner shown in FIG.

  Subsequently, the CPU 26 instructs the driver 18c to repeat the exposure operation and the charge reading operation under the face registration task in order to execute the moving image capturing process. In response to a vertical synchronization signal Vsync periodically generated from an SG (Signal Generator) (not shown), the driver 18c exposes the imaging surface and reads out the charges generated on the imaging surface in a raster scanning manner. From the imager 16, raw image data based on the read charges is periodically output.

  The preprocessing circuit 20 performs processing such as digital clamping, pixel defect correction, and gain control on the raw image data output from the imager 16. The raw image data subjected to these processes is written into the raw image area 32 a of the SDRAM 32 through the memory control circuit 30.

  The post-processing circuit 34 reads the raw image data stored in the raw image area 32a through the memory control circuit 30, and performs color separation processing, white balance adjustment processing, and YUV conversion processing on the read raw image data. The post-processing circuit 34 further performs display zoom processing and search zoom processing in parallel on the image data in the YUV format. As a result, display image data and search image data conforming to the YUV format are individually created. The display image data is written into the display image area 32 b of the SDRAM 32 by the memory control circuit 30. The search image data is written into the search image area 32 c of the SDRAM 32 by the memory control circuit 30.

  The LCD driver 36 repeatedly reads the display image data stored in the display image area 32b through the memory control circuit 30, and drives the LCD monitor 38 based on the read image data. As a result, a real-time moving image (through image) of the object scene is displayed on the monitor screen.

  Referring to FIG. 4, an evaluation area EVA is assigned to the center of the imaging surface. The evaluation area EVA is divided into 16 in each of the horizontal direction and the vertical direction, and 256 divided areas form the evaluation area EVA. In addition to the above-described processing, the preprocessing circuit 20 shown in FIG. 2 executes simple RGB conversion processing that simply converts raw image data into RGB data.

  The AE evaluation circuit 22 integrates RGB data belonging to the evaluation area EVA among the RGB data generated by the preprocessing circuit 20 every time the vertical synchronization signal Vsync is generated. As a result, 256 integral values, that is, 256 AE evaluation values, are output from the AE evaluation circuit 22 in response to the vertical synchronization signal Vsync.

  The AF evaluation circuit 24 integrates the high-frequency components of the RGB data belonging to the evaluation area EVA among the RGB data generated by the preprocessing circuit 20 every time the vertical synchronization signal Vsync is generated. As a result, 256 integral values, that is, 256 AF evaluation values, are output from the AF evaluation circuit 24 in response to the vertical synchronization signal Vsync.

  When the shutter button 28sh is in the non-operating state, the CPU 26 executes a simple AE process based on the output from the AE evaluation circuit 22 under the face registration task, and calculates an appropriate EV value. The aperture amount and exposure time that define the calculated appropriate EV value are set in the drivers 18b and 18c, respectively, and as a result, the brightness of the through image is adjusted appropriately.

  The CPU 26 also searches for a human face image from the search image data stored in the search image area 32c under a registration face detection task executed in parallel with the face registration task. For such a registration face detection task, a standard face dictionary STDC shown in FIG. 5, a face frame register RGST1 shown in FIG. 6, and a plurality of face detection frames FD, FD, FD,. . The standard face dictionary STDC, the face frame register RGST1, and the plurality of face detection frames FD, FD, FD,... Are also used in an imaging face detection task described later.

  Each time the vertical synchronization signal Vsync is generated, the face detection frame FD moves on the search image area 32b in a raster scanning manner corresponding to the registration frame RF1 (see FIG. 8). The size of the face detection frame FD is reduced in increments of “5” from “70” to “40” every time raster scanning ends.

  The CPU 26 reads the image data belonging to the face detection frame FD from the search image area 32b through the memory control circuit 30, and calculates the feature amount of the read image data. The calculated feature amount is collated with the feature amount of the face image registered in the standard face dictionary. When the matching degree exceeds the reference value REF, it is considered that a face image has been found from the registration frame RF1, and the variable CNT is updated from “0” to “1”. Further, the current position and size of the face detection frame FD are registered in the face frame register RGST1 as the position and size of the face frame surrounding the found face image.

  When the shutter button 28sh is in a non-operation state, the CPU 26 repeatedly determines the value of the variable CNT under the face registration task. If the variable CNT is “0”, a face frame non-display command is issued to the graphic generator 46, and if the variable CNT is “1”, a face frame display command is issued to the graphic generator 46. The face frame display command describes the position and size of the face frame registered in the face frame register RGST1.

  When the face frame display command is given, the graphic generator 46 creates graphic data representing the face frame KF1, and gives the created graphic data to the LCD driver 36. The graphic data is created with reference to the position and size described in the face frame display command. As a result, the face frame KF1 is displayed so as to surround the found face image.

  When the face image of “Kenji” appears in the registration frame RF1, the face frame KF1 is displayed as shown in FIG. When the face image “Daisuke” appears in the registration frame RF1, the face frame KF1 is displayed as shown in FIG. 9B. Further, when the face image of “History” appears in the registration frame RF1, the face frame KF1 is displayed in the manner shown in FIG. 9C. When the face image “Keiko” appears in the registration frame RF1, the face frame KF1 is displayed as shown in FIG.

  When the shutter button 28sh is operated while the face frame KF1 is displayed, the CPU 26 executes a still image capturing process under the face registration task. One frame of display image data representing the object scene at the time when the shutter button 28sh is operated is captured into the still image area 32d by the still image capturing process. Thereafter, the CPU 26 extracts a part of the image data belonging to the face frame KF1 from the image data captured in the still image area 32d. Thereby, a registered face image is obtained.

  Subsequently, the CPU 26 gives a registration operation screen display command to the graphic generator 46 under the face registration task. The graphic generator 46 creates graphic data representing the registration operation screen and gives the created graphic data to the LCD driver 36. As a result, the registration operation screen shown in FIG. 10 is displayed on the LCD monitor 38. According to FIG. 10, the registration operation screen is formed by “registration” and “cancel” icons and a person name input field.

  When a desired person name is entered in the person name input field and then the “register” icon is selected, the CPU 26 considers that the name input operation has been performed, and determines the smile level of the registered face image extracted in the manner described above. calculate. The extracted registered face image is described in the registered face dictionary RGDC together with the name of the person and the smile level.

  Therefore, when the shutter button 28sh is operated in the state shown in FIG. 9A, a registered face image showing the face of “Kenji” is created. When “Kenji” is input by the name input operation, the smile level (= 70) of the registered face image is calculated. The registered face image, person name (= Kenji) and smile level (= 70) are described in the first column of the registered face dictionary RGDC shown in FIG.

  When the shutter button 28sh is operated in the state shown in FIG. 9B, a registered face image showing the face of “Daisuke” is created. When “Daisuke” is input by the name input operation, the smile level (= 100) of the registered face image is calculated. The registered face image, personal name (= Daisuke), and smile level (= 100) are described in the second column of the registered face dictionary RGDC shown in FIG.

  When the shutter button 28sh is operated in the state shown in FIG. 9C, a registered face image representing the face of “History” is created. When “History” is input by the name input operation, the smile level (= 0) of the registered face image is calculated. The registered face image, personal name (= History) and smile level (= 0) are described in the third column of the registered face dictionary RGDC shown in FIG.

  When the shutter button 28sh is operated in the state shown in FIG. 9D, a registered face image showing the face of “Keiko” is created. When “Keiko” is input by the name input operation, the smile level (= 0) of the registered face image is calculated. The registered face image, personal name (= Keiko), and smile level (= 0) are described in the fourth column of the registered face dictionary RGDC shown in FIG.

  Although the entire head is drawn as a face image according to FIG. 11, strictly speaking, only images belonging to the face frame KF1 are stored in the registered face dictionary RGDC as face images.

  When the imaging mode is selected instead of the face registration mode, the CPU 26 starts moving image capturing processing under the imaging task. Thereby, the through image is displayed on the LCD monitor 38, and the search image data is repeatedly written in the search image area 32c.

  When the shutter button 28sh is in the non-operating state, the CPU 26 executes a simple AE process based on the output from the AE evaluation circuit 22 under the imaging task, and calculates an appropriate EV value. The aperture amount and exposure time that define the calculated appropriate EV value are set in the drivers 18b and 18c, respectively, and as a result, the brightness of the through image is adjusted appropriately.

  The CPU 26 searches for a human face image from the search image data stored in the search image area 32c under an imaging face detection task executed in parallel with the imaging task. The searched face image is an image that matches the registered face image stored in the registered face dictionary RGDC.

  The face detection frame FD shown in FIG. 7 moves on the search image area 32b in a raster scanning manner corresponding to the evaluation area EVA every time the vertical synchronization signal Vsync is generated (see FIG. 12). The size of the face detection frame FD is reduced in increments of “5” from “200” to “20” every time raster scanning ends.

  The CPU 26 reads the image data belonging to the face detection frame FD from the search image area 32b through the memory control circuit 30, and calculates the feature amount of the read image data. The calculated feature amount is collated with the feature amount of the face image stored in the standard face dictionary STDC. When the matching degree exceeds the reference value REF, the variable CNT having “0” as an initial value is incremented. Further, the current position and size of the face detection frame FD are set in the face frame register RGST1 shown in FIG. 6 as the position and size of the face frame surrounding the face image.

  Therefore, when the object scene shown in FIG. 13 is captured, the human H1, H2 and Kenji face images are detected, and the position and size of the face frame surrounding the detected face image are set in the face frame register RGST1. The The position and size of the face frame corresponding to the person H1 are described in the first column of the face frame register RGST1, the position and size of the face frame corresponding to the person H2 are described in the second column of the face frame register RGST1, The position and size of the face frame corresponding to Kenji are described in the third column of the face frame register RGST1. When the position and size of the face frame corresponding to the constitution are described, the variable CNT indicates “3”.

  Subsequently, the CPU 26 sequentially specifies the CNT face frames set in the face frame register RGST1. Image data belonging to the designated face frame is subjected to the following face recognition process.

  First, image data belonging to the designated face frame is read from the search image area 32c through the memory control circuit 30, and the smile level of the read image data is calculated. Next, the variable K is set to each of “1” to “Kmax”, and the smile level of the registered face image described in the Kth column is detected from the registered face dictionary RGDC. “Kmax” corresponds to the total number of registered face images.

  The feature amount of the image data belonging to the designated face frame is corrected so that the difference between the calculated and detected two smile levels is suppressed or eliminated. When the feature amount of the Kenji face image shown in FIG. 14A is corrected corresponding to K = 1, it is described in the smile degree (= 0) of the face image and the registered face dictionary RGDC shown in FIG. The smile level (= 70) of Kenji's face image is referred to. As a result of correcting the feature amount so as to eliminate these differences in smile levels, the facial expression of Kenji's face image changes from FIG. 14 (A) to FIG. 14 (B).

  The feature amount thus corrected is collated with the feature amount of the registered face image described in the Kth column. If the matching level exceeds the reference value REF, the column number (= K = registered face image number) of the registered face image as the matching destination and the matching level are registered in the recognition face register RGST2 shown in FIG.

  If at least one registered face image number is set in the recognition face register RGST2, the flag FLG_RCG is set to “1”, the registered face image number corresponding to the maximum matching degree, the position and size of the designated face frame, Is registered in the confirmation register RGST3 shown in FIG. On the other hand, if the registered face image number does not exist in the recognized face register RGST2, the flag FLG_RCG is set to “0”.

  When the shutter button 28sh is not operated, the CPU 26 repeatedly determines the value of the flag FLG_RCG under the imaging task. If the flag FLG_RCG is “0”, a face frame non-display command and a personal name non-display command are issued to the graphic generator 46. If the flag FLG_RCG is “1”, a face frame display command and a personal name display command are issued to the graphic generator 46. It is issued towards. The face frame display command describes the position and size of the face frame set in the confirmation register RGST3. In the personal name display command, the personal name corresponding to the registered face image number set in the confirmation register RGST3 is described.

  When the face frame display command is given, the graphic generator 46 creates graphic data representing the face frame KF1, and gives the created graphic data to the LCD driver 36. The graphic data is created with reference to the position and size described in the face frame display command. As a result, the face frame KF1 is displayed so as to surround the found face image.

  When the personal name display command is given, the graphic generator 46 creates graphic data representing the personal name and supplies the created graphic data to the LCD driver 36. Your name is displayed in the lower right corner of the monitor screen.

  Therefore, when the object scene shown in FIG. 13 is captured, the face frame KF1 and the name of the person are displayed on the monitor screen as shown in FIG.

  When the shutter button 28sh is half-pressed, the CPU 26 executes a strict AE process based on the output from the AE evaluation circuit 22 under the imaging task, and calculates an optimum EV value. The aperture amount and the exposure time that define the optimum EV value are set in the drivers 18b and 18c, respectively. As a result, the brightness of the through image is appropriately adjusted. When the strict AE process is completed, the CPU 26 executes the AF process under the imaging task. The AF process is executed by a hill-climbing method with reference to the output of the AF evaluation circuit 24, and the focus lens 12 is set at the focal point. This improves the sharpness of the live view image.

  When the shutter button 28sh is fully pressed, a still image capturing process and a recording process are executed. One frame of display image data at the time when the shutter button 28sh is fully pressed is captured into the still image area 32d by the still image capturing process. The captured one-frame image data is read from the still image area 32d by the I / F 40 activated in association with the recording process, and is recorded on the recording medium 42 in a file format.

  The CPU 26 performs the main task shown in FIG. 18, the face registration task shown in FIGS. 19 to 21, the registration face detection task shown in FIGS. 22 to 24, the imaging task shown in FIGS. 25 to 27, and FIGS. A plurality of tasks including the imaging face detection task shown in FIG. Note that control programs corresponding to these tasks are stored in the flash memory 44.

  Referring to FIG. 18, in step S1, it is determined whether or not the current operation mode is the face registration mode, and in step S3, it is determined whether or not the current operation mode is the imaging mode. If YES in step S1, the face registration task is activated in step S5, and if YES in step S3, the imaging task is activated in step S7. If both steps S1 and S3 are NO, other processing is executed in step S9. When the process of step S5, S7 or S9 is completed, it is repeatedly determined in step S11 whether or not a mode switching operation has been performed. When the determination result is updated from NO to YES, the activated task is stopped in step S13, and thereafter, the process returns to step S1.

  Referring to FIG. 19, a registration frame display command is given to graphic generator 46 in step S21. As a result, the registration frame RF1 is displayed on the LCD monitor 38. In step S23, a moving image capturing process is executed. As a result, a through image representing the scene is displayed on the LCD monitor 38, and search image data is repeatedly written in the search image area 32c.

  In step S25, the variable W is set to “0”, and in step S27, the registration face detection task is activated. The variable W is a variable for identifying whether or not a face image has been found by the registration face detection task. “0” indicates undiscovered, while “1” indicates discovery.

  In step S29, it is determined whether or not the flag FLG_FIN indicates “1”. The flag FLG_FIN is set to “0” at the start of the registration face detection task, and is updated to “1” at the end of the registration face detection task. If the determination result is YES, the process proceeds to step S39, and if the determination result is NO, the process proceeds to step S31.

  In step S39, the value of variable CNT is determined. The variable CNT is a variable indicating the number of face images discovered by the registration face detection task, and indicates either “0” or “1”. If the variable CNT is “1”, a face frame display command is given to the graphic generator 46 in step S41, and the variable W is set to “1” in step S43. If the variable CNT is “0”, a face frame non-display command is given to the graphic generator 46 in step S45, and the variable W is set to “0” in step S47. As a result of the processing in step S41, the face frame KF1 is displayed so as to surround the face image that has been found. Further, as a result of the processing in step S45, the display of the face frame KF1 is stopped. When the process of step S43 or S47 is completed, the process returns to step S27.

  In step S31, it is determined whether or not the shutter button 28sh has been operated. In step S33, it is determined whether or not the variable W indicates “1”. If the determination result in step S31 or S33 is NO, a simple AE process is executed in step S35, and then the process returns to step S29. On the other hand, if both the determination results in steps S31 and S33 are YES, a still image capturing process is executed in step S37, and then the process proceeds to step S51.

  Through the simple AE process, the brightness of the through image is appropriately adjusted. In addition, one frame of image data representing the scene at the time when the shutter button 28 sh is operated is captured into the still image area 32 d by the still image capturing process.

  In step S51, a part of the image data belonging to the face frame RF1 is extracted as a registered face image. When the extraction process is completed, the process proceeds to step S53, and a registration operation screen display command is given to the graphic generator 46. As a result, a registration operation screen is displayed on the LCD monitor 38. In step S55, it is determined whether or not a name input operation has been performed. In step S57, it is determined whether or not a cancel operation has been performed.

  If “YES” in the step S55, the process proceeds to a step S59, and the smile degree of the registered face image extracted in the step S51 is calculated. In step S61, the registered face image extracted in step S51, the person name input by the name input operation, and the smile level calculated in step S59 are stored in the registered face dictionary RGDC. When the process of step S61 is completed, the variable W is set to “0” in step S63, and then the process returns to step S27. On the other hand, if “YES” in the step S57, the process returns to the step S27 through the process of the step S63.

  Referring to FIG. 22, in step S71, flag FLG_FIN is set to “0”, and in step S73, the inside of registration frame RF1 is set as a search area. In step S75, the maximum size SZmax is set to “70” and the minimum size SZmin is set to “40” in order to define a variable range of the size of the face detection frame FD. When the definition of the variable range is completed, the variable CNT is set to “0” in step S77, and the size of the face detection frame FD is set to “SZmax” in step S79.

  In step S81, it is determined whether or not the vertical synchronization signal Vsync has been generated. If the determination result is updated from NO to YES, the face detection frame FD is arranged in the upper left position of the search area in step S83. In step S85, a part of the search image data belonging to the face detection frame FD is read from the search image area 32c, and the feature amount of the read search image data is calculated.

  In step S87, the calculated feature amount is collated with the feature amount of the face image stored in the standard face dictionary STDC. In step S89, it is determined whether or not the collation degree exceeds the reference value REF. If the determination result is YES, the process proceeds to step S91, and if the determination result is NO, the process proceeds to step S97.

  In step S91, the variable CNT is set to “1”. In step S93, the current position and size of the face detection frame FD are registered in the face frame register RGST1 as the position and size of the face image. When the registration is completed, the flag FLG_FIN is set to “1” in step S95, and then the process ends.

  In step S97, it is determined whether or not the face detection frame FD has reached the lower right position of the search area. If the determination result is NO, the face detection frame FD is moved in the raster direction by a predetermined amount in step S99, and then the process returns to step S85. If the determination result is YES, it is determined in a step S101 whether or not the size of the face detection frame FD is “SZmin” or less. If the determination result is NO, the size of the face detection frame FD is reduced by “5” in step S103, the face detection frame FD is placed in the upper left position of the search area in step S105, and then the process returns to step S85. If the determination result of step S101 is YES, it will progress to step S95.

  Referring to FIG. 25, in step S111, a moving image capturing process is executed, and in step S113, an imaging face detection task is activated. In step S115, it is determined whether or not the flag FLG_FIN indicates “1”. If the determination result is YES, the process proceeds to step S125, and if the determination result is NO, the process proceeds to step S117.

  In step S125, it is determined whether or not the flag FLG_RCG is “1”. If the determination result is YES, a face frame display command is given to the graphic generator 46 in step S127, and a personal name display command is given to the graphic generator 46 in step S129. On the other hand, if the determination result is NO, a face frame non-display command is given to the graphic generator 46 in step S131, and a personal name non-display command is given to the graphic generator 46 in step S133.

  As a result of the processing in step S127, the face frame KF1 is displayed so as to surround the face image that matches the registered face image set in the confirmation register RGST3. In addition, as a result of the processing in step S129, the person name corresponding to the registered face image set in the confirmation register RGST3 is displayed. Further, as a result of the processing in steps S131 to 133, the display of the face frame KF1 and the person name is stopped. When the process of step S129 or S133 is completed, the process returns to step S113.

  In step S117, it is determined whether or not the shutter button 28sh is half-pressed. If the determination result is NO, the simple AE process is executed in step S119, and if the determination result is YES, the strict AE process is executed in step S121. The brightness of the through image is moderately adjusted by the simple AE process and adjusted to the optimum value by the strict AE process. When the process of step S119 is completed, the process returns to step S115, and when the process of step S121 is completed, the AF process is executed in step S123. The focus lens 12 is placed at the focal point by AF processing.

  In step S135, it is determined whether or not the shutter button 28sh is fully pressed. In step S137, it is determined whether or not the operation of the shutter button 28sh is released. If “YES” in the step S135, a still image capturing process is executed in a step S139, a recording process is executed in a step S141, and then the process returns to the step S113. If “YES” in the step S137, the process returns to the step S113 as it is. As a result of the processing in step S139, one frame of image data representing the object scene at the time when the shutter button 28sh is fully pressed is taken into the still image area 32d. In addition, as a result of the processing in step S141, the image data captured in the still image area 32d is recorded on the recording medium 42 in a file format.

  Referring to FIG. 28, in step S151, flag FLG_FIN is set to “0”, and in step S153, the entire evaluation area EVA is set as a search area. In step S155, in order to define a variable range of the size of the face detection frame FD, the maximum size SZmax is set to “200”, and the minimum size SZmin is set to “20”. When the definition of the variable range is completed, the variable CNT is set to “0” in step S157, and the size of the face detection frame FD is set to “SZmax” in step S159.

  In step S161, it is determined whether or not the vertical synchronization signal Vsync is generated. If the determination result is updated from NO to YES, the face detection frame FD is arranged in the upper left position of the search area in step S163. In step S165, a part of the search image data belonging to the face detection frame FD is read from the search image area 32c, and the feature amount of the read search image data is calculated.

  In step S167, the calculated feature amount is compared with the feature amount of the face image stored in the standard face dictionary STDC, and in step S169, it is determined whether or not the matching degree exceeds the reference value REF. If a determination result is NO, it will progress to step S175 as it is, and if a determination result is YES, it will progress to step S175 through step S171 and S173. In step S171, the variable CNT is incremented. In step S173, the current position and size of the face detection frame FD are registered in the face frame register RGST1 as the position and size of the face frame surrounding the face image.

  In step S175, it is determined whether or not the face detection frame FD has reached the lower right position of the search area. If the determination result is NO, the face detection frame FD is moved in the raster direction by a predetermined amount in step S177, and then the process returns to step S165. If the determination result is YES, it is determined in a step S179 whether or not the size of the face detection frame FD is “SZmin” or less. If the determination result is NO, the size of the face detection frame FD is reduced by “5” in step S181, the face detection frame FD is placed in the upper left position of the search area in step S183, and then the process returns to step S165. If the determination result of step S179 is YES, it will progress to step S185.

  In step S185, the variable N is set to “1”, and in step S187, it is determined whether or not the variable N exceeds the variable CNT. If the determination result is NO, the process proceeds to step S189, and the face frame set in the Nth column of the face frame register RGST1 is designated. In step S191, face recognition processing focusing on image data belonging to the designated face frame is executed. When the process is completed, the variable N is incremented in step S193, and then the process returns to step S187. If the decision result in the step S187 is YES, a flag FLG_FIN is set to “1” in a step S195, and then the process is ended.

  The face recognition process in step S191 is executed according to a subroutine shown in FIGS. First, the smile level of the image data belonging to the designated face frame is calculated in step S201, the recognition face register RGST2 is cleared in step S203, and the variable K is set to “1” in step S205. In step S207, it is determined whether or not the variable K exceeds a maximum value Kmax (= total number of registered face images). If the determination result is NO, the process proceeds to step S217, and the smile level of the Kth registered face image is detected from the registered face dictionary RGDC. In step S219, the feature amount of the image data belonging to the designated face frame is corrected so that the difference between the smile level calculated in step S201 and the smile level detected in step S217 is suppressed or eliminated.

  In step S221, the corrected feature amount is collated with the feature amount of the Kth registered face image. In step S223, it is determined whether or not the collation degree exceeds the reference value REF. If the determination result is YES, the process proceeds to step S225, and the registered face image number (= K) indicating the registered face image of the collation destination and the collation degree are registered in the recognition face register RGST2. When registration is completed, the variable K is incremented in step S227, and then the process returns to step S207. If the determination result of step S223 is NO, the process returns to step S207 through the process of step S227.

  If the determination result in step S207 is YES, it is determined in step S209 whether or not at least one registered face image number is set in the recognition face register RGST2. If the determination result is YES, a flag FLG_RCG is set to “1” in a step S211. In step S213, the registered face image number corresponding to the maximum matching degree and the position and size of the designated face frame are registered in the confirmation register RGST3. If the determination result of step S209 is NO, a flag FLG_RCG is set to “0” in step S215. When the process of step S213 or S215 is completed, the process returns to the upper hierarchy routine.

  As can be seen from the above description, the CPU 26 searches for one or more face images from the search image data stored in the search image area 32c of the SDRAM 32 (S153 to S183), and sequentially finds the face images thus found. Specify (S189). The CPU 26 also detects the facial expression of the designated face image (S201), and searches for a registered facial image that matches the designated facial image among a plurality of registered facial images with reference to the detected facial expression (S203 to S203). 207, S217 to S227). Thereafter, the CPU 26 displays attribute information indicating the attribute of the registered face image found on the LCD monitor 38 (S209 to S215, S125 to S133).

  When searching for a registered face image that matches the face image that appears in the search image data, the facial expression of the face image that appears in the search image data is referred to. The attribute information displayed on the LCD monitor 38 indicates the attribute of the registered face image thus discovered, that is, the attribute of the face image that appears in the object scene image. As a result, the operating performance is improved.

  In this embodiment, in order to suppress or eliminate the difference between the smile level calculated in step S201 and the smile level detected in step S217, the feature amount of the face image belonging to the designated face frame is corrected. ing. However, the feature amount of the Kth registered face image may be corrected in place of the feature amount of the face image belonging to the designated face frame or together with the feature amount of the face image belonging to the designated face frame. Alternatively, the target to be corrected may be a face image instead of a feature amount, and the feature amount of the corrected face image may be detected.

  In this embodiment, in order to correct the feature amount, the smile level of the face image belonging to the designated face frame and the smile level of the Kth registered face image are obtained. However, instead of this, the face image belonging to the designated face frame and / or the Kth registered face image is repeatedly corrected so as to show different smile levels, and the feature amount of the face image belonging to the designated face frame and the Kth The feature amount of the registered face image may be collated for each correction process, and the registered face image that matches the face image belonging to the designated face frame may be specified based on such a collation result.

  In this case, a registered face dictionary RGDC shown in FIG. 34 is prepared instead of the registered face dictionary RGDC shown in FIG. 11, and the face recognition processes shown in FIGS. 35 to 37 are substituted for the face recognition processes shown in FIGS. Need to run.

  According to FIG. 34, a normal (smiling degree = 0) face image is described in each column forming the registered face dictionary RGDC. According to FIG. 35, the processing corresponding to step S201 shown in FIG. 32 is omitted. 35 is the same as the process of steps S203 to S215 shown in FIG. 32, and thus a duplicate description is omitted.

  If the determination result in step S207 is NO, the variable L is set to “1” in step S231, and the smile level is set to “L * 10” in step S233. In step S235, the feature amount of the registered face image described in the Kth column is corrected with reference to the smile level set in step S233. In step S237, the feature amount of the image data belonging to the designated face frame is compared with the feature amount corrected in step S237, and in step S239, it is determined whether or not the variable L has reached “10”.

  If the determination result is NO, the variable L is incremented in step S241 and then the process returns to step S233. If the determination result is YES, the process proceeds to step S243. In step S243, the maximum matching degree is detected from the total of ten matching degrees obtained by the process in step S237, and in step S245, it is determined whether or not the detected maximum matching degree exceeds the reference value REF. If the determination result is YES, the process proceeds to step S247, and the registered face image number (= K) indicating the registered face image of the collation destination and the maximum collation degree are registered in the recognition face register RGST2. When registration is completed, the variable K is incremented in step S249, and then the process returns to step S207. If the determination result of step S245 is NO, the process returns to step S207 through the process of step S249.

  Further, in the above-described embodiment, in order to suppress or eliminate the difference between the smile level of the face image belonging to the designated face frame and the smile level of the Kth registered face image, the face image correction process is executed. Yes. However, a plurality of registered face images representing a common face with different smile levels are prepared in each column of the registered face dictionary RGDC, and each registered face image having a smile level close to that of the face image belonging to the designated face frame is set. The registered face image selected from the column may be collated with the face image belonging to the designated face frame.

  In this case, the registered face dictionary RGDC shown in FIG. 38 is prepared instead of the registered face dictionary RGDC shown in FIG. 11, and the face recognition process shown in FIGS. 39 to 41 is substituted for the face recognition process shown in FIGS. Need to run.

  According to FIG. 38, in each column forming the registered face dictionary RGDC, a face image of a common person is described corresponding to a normal face (smile level = 0) and a smile (smile level = 100).

  According to FIG. 39, first, the smile level of the image data belonging to the designated face frame is calculated in step S251. In step S253, it is determined whether or not the calculated smile level is less than a threshold TH (= 50). If the determination result is YES, the normal face is set as a reference face in step S255, while the determination result is NO. If there is, the smile is set as a reference face in step S257. In step S259, the recognition face register RGST2 is cleared, and in step S261, the variable K is set to “1”. In step S263, it is determined whether or not the variable K exceeds the maximum value Kmax. If the determination result is NO, the process proceeds to step S273.

  In step S273, the feature amount of the image data belonging to the designated face frame is collated with the feature amount of the Kth registered face image. In step S275, it is determined whether or not the collation degree exceeds the reference value REF. If the determination result is YES, the process proceeds to step S277 to register the registered face image number (= K) indicating the registered face image of the collation destination and the collation degree in the recognition face register RGST2. When registration is completed, the variable K is incremented in step S279, and then the process returns to step S263. If the determination result of step S275 is NO, the process returns to step S263 through the process of step S279.

  If the determination result in step S263 is YES, it is determined in step S265 whether or not at least one registered face image number is set in the recognition face register RGST2. If the determination result is YES, a flag FLG_RCG is set to “1” in a step S267. In step S269, the registered face image number corresponding to the maximum matching degree and the position and size of the designated face frame are registered in the confirmation register RGST3. If the determination result of step S265 is NO, the flag FLG_RCG is set to “0” in step S271. When the process of step S269 or S271 is completed, the process returns to the upper hierarchy routine.

DESCRIPTION OF SYMBOLS 10 ... Digital camera 16 ... Imager 22 ... AE evaluation circuit 24 ... AF evaluation circuit 26 ... CPU
32 ... SDRAM
44 ... Flash memory

Claims (10)

First search means for searching for one or more face images from the object scene image;
First designation means for designating each of one or more face images discovered by the first search means as a first target face image;
A second search means for searching a registered face image that matches the first noticeable face image from a plurality of registered face images with reference to a facial expression of the first noticeable face image; A face search device comprising output means for outputting attribute information indicating an attribute of a registered face image.   The second search means designates each of the plurality of registered face images as a second attention face image, and the difference between the expression of the first attention face image and the expression of the second attention face image The face search device according to claim 1, further comprising a correcting unit that corrects the first attention face image and / or the second attention face image so that the image is suppressed. Each of the plurality of registered face images includes a plurality of facial expression parameters,
The face search according to claim 2, wherein the second search means further includes an expression parameter selection means for selecting an expression parameter of the second attention face image from the plurality of expression parameters for the correction process of the correction means. apparatus.   The second searching means is a second specifying means for specifying each of the plurality of registered face images as a second attention face image, and the first attention face image and / or the second attention face so as to show different expressions. Correction means for repeatedly correcting an image; collation means for comparing the first attention face image and the second attention face image for each correction processing by the correction means; and a registered face image that matches the first attention face image The face search device according to claim 1, further comprising a specifying unit that specifies based on a collation result of the collating unit. Each of the plurality of registered face images includes a plurality of images representing a common face with different facial expressions,
The second search means selects an image having a facial expression along the facial expression of the first attention face image from the plurality of images in association with the designation process of the second designation means, and the image The face search apparatus according to claim 1, further comprising a matching unit that matches an image selected by the selection unit with the first target pixel image. Each of the plurality of registered face images has a plurality of names,
The face search device according to claim 1, wherein the attribute information output by the output unit includes a name of a registered face image found by the second search unit. It further includes an imaging means for capturing the scene,
The face search device according to claim 1, wherein the first search unit performs a search process on the object scene image output from the imaging unit.   The face search device according to claim 1, wherein the first search means executes a search process with reference to a standard face image. In the processor of the face search device,
A first search step for searching for one or more face images from the object scene image;
A designating step of designating each of the one or more face images discovered by the first search step as a first target face image;
A second search step of searching for a registered face image matching the first target face image from a plurality of registered face images with reference to a facial expression of the first target face image; and A face search program for executing an output step of outputting attribute information indicating an attribute of a registered face image. A face search method executed by a face search device,
A first search step for searching for one or more face images from the object scene image;
A designating step of designating each of the one or more face images discovered by the first search step as a first target face image;
A second search step of searching for a registered face image matching the first target face image from a plurality of registered face images with reference to a facial expression of the first target face image; and A face search method comprising an output step of outputting attribute information indicating an attribute of a registered face image.

Images