JP5484994B2 - Facial image synthesis device, facial image synthesis method, and program - Google Patents

Description

The present invention relates to a face image composition device, a face image composition method, and a program. More specifically , the present invention relates to a face image composition device and a face image composition method capable of restoring a failed image taken outdoors or in a backlit state or with a large difference in shadow. And related to the program.

  In recent years, portable devices (photographing devices) with a photographing function such as a digital camera have improved image processing functions and have overcome various poor photographing scenes. For example, in dark or backlit scenes, the sensitivity and gain of the imaging unit are increased, and the sensitivity and gain are partially switched to perform processing that was impossible in the era of film cameras and prevent failures. doing. In addition, improvement in image processing technology makes it possible to obtain a proper image by correcting the brightness of an image, even if the image has failed due to insufficient exposure.

  Furthermore, recently, a photographing apparatus has been proposed in which a photograph desired by a photographer is obtained by combining images. For example, Patent Document 1 proposes an image processing method that corrects a captured image by performing image processing that opens eyes on a person image with closed eyes. That is, it is disclosed that the eye portion of a person image with closed eyes is replaced with an open eye image.

JP 2001-197296 A

  As described above, an apparatus and a method for correcting a failed image by image synthesis have been proposed, but there is no disclosure of obtaining an appropriate image from a captured image with poor exposure using image synthesis. In other words, in situations where there is a significant difference in brightness between the background and the face image, such as blackout or overexposure, information such as shading on the face is lost, and image processing that corrects the brightness of the image So you can't help. On the other hand, image composition has been proposed in which images such as the face and eyes in a photographed image are replaced with images prepared in advance, but it does not relieve a photographed image with poor exposure.

The present invention has been made in view of such circumstances, and it is possible to easily restore a captured image in which information regarding shadows of a portion such as a face in the captured image is lost, such as blackout or overexposure. An object of the present invention is to provide a face image synthesizing device, a face image synthesizing method, and a program.

The face image synthesizer according to the first aspect of the present invention is a face image database that stores an image including a face portion, and an appropriateness for determining whether the image is appropriate based on the brightness of the face portion in the photographed image. A first correction unit that corrects the captured image when the image is determined to be a failure image by the image determination unit and the appropriate image determination unit; If the image determination unit determines that the image is a failed image, and if the image cannot be made an appropriate image by correcting the brightness of the captured image, it matches or is similar to the face in the failed image from the face image database A second correction unit that searches for a face image to be corrected, corrects the brightness of the searched face image, and an image for performing image synthesis that replaces the face image corrected by the second correction unit with the face image of the failed image. A combining unit, the.

In the face image synthesizing apparatus according to the second invention, in the first invention, the appropriate image determination unit generates a histogram of the photographed image and makes a determination based on the histogram.
According to a third aspect of the present invention, in the first aspect of the invention, the appropriate image determination unit makes a determination based on the difference between the average brightness of the face portion of the photographed image and the brightness of the background. .
In the face image composition device according to a fourth aspect of the present invention, in the first aspect , the appropriate image determination unit makes a determination based on a difference in brightness in the face portion of the photographed image.

In the face image synthesizing device according to a fifth aspect of the present invention, in the first aspect , the second correction unit is configured to obtain an average of the face portion of the photographed image based on the brightness of the background. The brightness is corrected to a brightness obtained by subtracting the correction amount based on the difference between the brightness of the background and the brightness of the background.

According to a sixth aspect of the present invention, in the first aspect , when the second correction unit determines that the image is a failure image based on a difference in brightness in the face portion of the photographed image. Corrects the searched face image so as to compress the difference in brightness in the face portion.

According to a seventh aspect of the present invention, there is provided the face image composition device according to the first aspect , wherein the composition unit uses the eye-nose position of the face image corrected by the second correction unit as the eye-nose position of the face image of the failure image. Replace the image to match.
Eighth face image correction and synthesizing apparatus according to the present invention, in the first aspect, the database for each person, the orientation of the expression and face storing different face images.

According to a ninth aspect of the present invention, there is provided a face image composition device according to the first aspect , further comprising a display unit for displaying an image and a display control unit for an image displayed on the display unit. Enlarges and displays the face portion of the photographed image, enlarges and displays the face image synthesized by the composition unit, and displays the original face of the photographed image including the synthesized face image. Reduce to size.

The face image composition method according to the tenth aspect of the present invention determines whether or not the image is appropriate based on the brightness of the face portion in the captured image, and determines that the image is a failure image by determining whether or not the image is appropriate. In this case, when the appropriate image is obtained by correcting the brightness of the captured image, the captured image is corrected and determined as a failed image by determining whether or not the image is the appropriate image, and the brightness of the captured image is corrected. If a face image matching the face in the failed image is searched from the face image database storing the image including the face portion when the image cannot be determined as an appropriate image by the brightness, the brightness of the searched face image Then, image synthesis is performed in which the face image corrected by the correction is replaced with the face image of the failed image.

A program according to an eleventh aspect of the invention is a computer program for performing face image synthesis in a computer having a face image database, and determines whether or not the image is appropriate based on the brightness of the face portion in the photographed image. And a step of correcting the captured image when the image is determined to be a failed image by determining whether or not the image is an appropriate image, and when the image becomes an appropriate image by correcting the brightness of the captured image; If it is determined as a failed image by determining whether or not a correct image cannot be obtained by correcting the brightness of the captured image, a face image that matches the face in the failed image is selected from the face image database. Searching, correcting the brightness of the searched face image, and correcting the face image corrected by the correction as the face image of the failed image. To execute a step of performing image combination changing can, to a computer.

According to the present invention, a face image composition device and a face image composition capable of easily restoring a photographed image in which information regarding shadows of a part of the photographed image such as a blackout or a whiteout is lost. Methods and programs can be provided.

It is a block diagram which shows mainly an electrical structure of the camera concerning 1st Embodiment of this invention. It is a figure explaining the synthesis | combination of the face image in the camera concerning 1st Embodiment of this invention. In the camera according to the first embodiment of the present invention, it is a diagram showing the luminance distribution of the subject, (a) shows the luminance distribution in the case of the overexposure subject, (b) shows the luminance distribution of the blackened subject. FIG. In the camera according to the first embodiment of the present invention, the luminance distribution of a subject is a modified example and is shown by a histogram, (a) shows the luminance distribution in the case of an overexposed subject, (b) shows It is a figure which shows the luminance distribution of the photographic subject. It is a flowchart which shows the operation | movement of face correction and a synthesis | combination of the camera concerning 1st Embodiment of this invention. In the camera according to the modification of the first embodiment of the present invention, it is a diagram showing the luminance distribution of the subject, (a) shows the luminance distribution in the case of overexposure subject, (b) is the subject of the blackened subject It is a figure which shows luminance distribution. It is a flowchart which shows operation | movement of the modification of the face correction and composition of the camera concerning the modification of 1st Embodiment of this invention. It is a figure explaining the synthesis | combination of the face image in the camera concerning 2nd Embodiment of this invention, (a) shows a mode when the whole face enters into the dark area | region, (b) is a face image of (a). Is an image synthesized using the reference data for complementation. It is a figure explaining the synthesis | combination of the face image in the camera concerning 2nd Embodiment of this invention, (a) shows the luminance distribution when the half of a face enters into the dark area | region, (b) is (a). The luminance distribution when a face image is synthesized using reference data for complementation is shown. In the camera according to the second embodiment of the present invention, the luminance distribution when the entire face is dark is shown, (a) is the luminance distribution before correction and synthesis, and (b) is the luminance after correction and synthesis. Show the distribution. In the camera according to the second embodiment of the present invention, the luminance distribution is shown when the face portion has a large luminance difference, (a) is the luminance distribution before correction and synthesis, and (b) is the luminance after correction and synthesis. Show the distribution. It is a flowchart which shows the operation | movement of face correction and a synthesis | combination of the camera concerning 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the unsuccessful face part determination of the camera concerning 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the failure degree determination of the camera concerning 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of correction | amendment of DB face image according to the failure degree of the camera concerning 2nd Embodiment of this invention. It is a figure which shows the display of the face part which carried out face correction and the synthesis | combination in the camera concerning the modification of 2nd Embodiment of this invention. It is a flowchart which shows the correction | amendment operation | movement of the synthesis | combination of a to-be-photographed object face image and correction | amendment DB face image in the camera concerning the modification of 2nd Embodiment of this invention. It is a figure explaining acquisition of the face image of face DB in the camera concerning a 2nd embodiment of the present invention. It is a figure explaining acquisition of the face image of face DB in the camera concerning a 2nd embodiment of the present invention. It is a figure which shows an example of the face image memorize | stored in face DB in the camera concerning 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of face image DB preparation of the camera concerning 2nd Embodiment of this invention. It is a block diagram which shows mainly an electrical structure of the camera concerning 3rd Embodiment of this invention.

  Hereinafter, preferred embodiments using a camera to which the present invention is applied will be described with reference to the drawings. A camera according to a preferred embodiment of the present invention is a digital camera, includes an imaging unit, converts a subject image into image data by the imaging unit, and converts the subject image into a main body based on the converted image data. Live view is displayed on the display unit placed on the back of the camera. The photographer determines the composition and the photo opportunity by observing the live view display. During the release operation, the image data is recorded in the recording unit. The image data recorded in the recording unit can be reproduced and displayed on the display unit when the reproduction mode is selected. In addition, when recording image data or reproducing recorded image data, if the face portion of the subject becomes a failed image due to backlight or the like, the face of the same person from the face database (DB) The image data is read out, the brightness is corrected so as not to cause a sense of incongruity with the surrounding brightness, and image synthesis is performed to replace the face portion of the failed image.

  The configuration of the camera 10 according to the first embodiment of the present invention will be described using the block diagram shown in FIG. The camera 10 includes a signal processing and control unit 1, an imaging unit 2, a face detection unit 3, a recording unit 4, a strobe unit 5, an operation determination unit 6, an audio recording unit 7, a display unit 8, a clock unit 9, a flash memory 11, The communication unit 12 is configured.

  The imaging unit 2 includes an exposure control unit such as a photographic lens and a shutter, an imaging device, a driving and reading circuit for the imaging device, and the like. The subject image formed by the photographic lens is converted into image data by the imaging device and is output. To do. In this specification, the image data is not limited to the image signal output from the image sensor, but is the image data processed by the signal processing and control unit 1, the image data recorded in the recording unit 4, and the like. Also used.

  The signal processing and control unit 1 includes a CPU (Central Processing Unit) and peripheral hardware circuits, and controls the entire sequence of the camera 10 according to a program stored in the flash memory 11. The signal processing and control unit 1 includes an exposure failure determination unit 1b. The poor exposure determination unit 1b analyzes the image data output from the imaging unit 2, and the subject's face is darker than the surroundings and underexposed and blacked out, or the face is brighter than the surroundings and overexposed and overexposed. And the shadow of the face of the subject is analyzed, and it is determined whether or not the exposure is poor because a part of the face is dark or bright. The signal processing and control unit 1 also performs various image processing such as image composition, face trimming, and correction processing.

  The face detection unit 3 detects whether or not a person's face is included in the subject image based on the image data output from the imaging unit 2. Face detection is performed based on whether or not there is a shadow pattern having human face characteristics in the image. If a face is included, the position of the face, the size of the face, the number of faces, etc. are also detected. Also, feature points necessary for determining whether or not they are the same person are extracted. Further, the eyes and mouth characteristics for determining facial expressions, the nose lines 27 (see FIG. 19) for determining the orientation of the face, and the like are also detected.

  The recording unit 4 includes a recording medium that can be attached to and detached from the camera body, or a built-in recording medium. The recording unit 4 records still image and moving image image data output from the imaging unit 2 and subjected to image processing by the signal processing and control unit 1, and accompanying data. The accompanying data includes the position, size, number, and the like of the face detected by the face detection unit 3 and shooting date / time information acquired from the clock unit 9 described later.

  In the face image database (face image DB) 4b in the recording unit 4, face image data is recorded for each individual such as a family or a friend. By using this face image DB 4b, it is possible to specify which person in the face image in the photographed image. The image data stored in the face image DB 4b stores face image data with different orientations and face images with different facial expressions for each individual. A failed image can be restored by replacing and compositing the face portion of the image data for the face image determined to be underexposed. The signal processing and control unit 1 described above can search the face image DB 4b for a human face image that matches the face portion of the captured image.

  The clock unit 9 has a time measuring function and outputs date / time information. At the time of shooting, as described above, the date / time information is recorded on the recording medium of the recording unit 4 together with the image data. The operation determination unit 6 includes various operation members such as a release button, a power button, a playback button, and a menu button arranged on the exterior of the camera 10, determines the operation state of the operation member, and performs image processing on the determination result. And output to the control unit 1.

  The display unit 8 is connected to the signal processing and control unit 1 and has a monitor such as a liquid crystal or an organic EL arranged on the back surface of the main body, and performs live view display, REC view display during recording, and recording. Display of recorded images recorded in the unit 4 and a control screen such as a menu screen are displayed.

  The strobe unit 5 is connected to the signal processing and control unit 1, and performs auxiliary illumination on the subject in response to a command from the signal processing and control unit 1. The signal processing and control unit 1 issues a light emission command for auxiliary illumination when the subject at the time of shooting is not in a brightness or shade suitable for shooting.

  The audio recording unit 7 includes a microphone, converts ambient audio into an audio signal, and outputs the audio signal to the signal processing and control unit 1. Audio data may be recorded together with image data when shooting a moving image. The flash memory 11 is a nonvolatile electrically rewritable memory, stores a program executed by the signal processing and control unit 1, and also stores various data necessary for controlling the camera 10. In addition, as long as a program etc. can be memorize | stored, other memory | storage devices, such as not only a flash memory but mask ROM, may be sufficient. The communication unit 12 communicates with external devices and accessories such as a personal computer by wireless or wired communication, and also communicates with an external server through the Internet or the like.

  Next, using FIG. 2, restoration by failure image correction and synthesis in the present embodiment will be described. FIG. 2A shows an example of a photographed image photographed by the camera 10. In this example, the subject A21 is properly exposed, but the subject B22 located behind is underexposed because it is not sufficiently bright. In this case, it is possible to correct the subject B22 so that the face portion of the subject B22 can be identified by performing partial image processing on the subject B22. For example, even in a state called blackout, it is possible to obtain an image without blackout by performing expansion or flattening in the histogram. Similarly, even if the face portion of the subject B22 is very bright and overexposed, it can be corrected by the same technique.

  However, in the case of extreme blackout, even if correction is performed by image processing, the amount of information in the original image is small, resulting in a noisy image as shown in FIG. In such a case, in this embodiment, as shown in FIG. 2C, the reference data for complementation of the face portion of the subject B22a photographed with appropriate exposure is used for the face portion of the subject B22. Thus, image composition for replacing the face portion of the subject B22 is performed.

  In other words, in a photographed image, the face is often the main subject, and there is a demand to prioritize exposure and focus over other parts. For this reason, recent photographing apparatuses are equipped with a face recognition device to recognize a face and adjust exposure and focus. In recent years, image processing technology has been remarkably advanced, and various image processing technologies for replacing a part of the original photographed image with another image have been proposed. Image processing for applying an image with open eyes has been proposed. In addition, image synthesis by affine transformation has become common, and application to face images has also been proposed. Therefore, in the present embodiment, using this face recognition and image processing technique, if the image is underexposed and an image with proper exposure cannot be obtained by correcting the brightness of the image, the image is synthesized by image synthesis. The image composition for replacing the part is performed.

  FIG. 3A is a luminance distribution diagram showing an exposure state (brightness), and for an example of a photographed image, the horizontal axis represents the luminance value and the vertical axis represents the frequency of brightness. An example of the photographed image shown in FIG. 3A is a case of underexposure, and there are many portions with low (dark) luminance values, so-called blackening. FIG. 3B is an example of another photographed image, which is a case of overexposure, in which there are many portions with high (bright) luminance values and so-called overexposure.

  The photographed image with the luminance distribution as shown in FIGS. 3A and 3B may not necessarily be a failed photograph depending on the photographer's intention and depending on the subject, but the photograph is intended for a person, a pet, or the like. In most cases, it is a failure photo. In this embodiment, the determination as to whether or not the photograph can be a failed photograph is made using the frequency U (see FIG. 3A) at a luminance value of zero in the case of blackout. In the case of overexposure, determination is made using the frequency O (see FIG. 3B) at the maximum luminance value. In the luminance distribution diagrams shown in FIGS. 3 (a) and 3 (b), there is a risk of erroneous determination due to noise. Therefore, as shown in FIGS. 4 (a) and 4 (b), a histogram with a rough luminance value width may be used. Good.

  Next, the face correction / combination operation in this embodiment will be described with reference to the flowchart shown in FIG. This flowchart and a flowchart described later are executed by the signal processing and control unit 1 in accordance with a program stored in the flash memory 11. This flow of face correction / combination is executed when the camera 10 captures an image and records the image data in the recording unit 4 or when the recorded image data is read and displayed.

  In the face correction / combination flow, first, a histogram of the first face is created (S1). Here, using the image data corresponding to the position of the face detected by the face detection unit 3 from the image data, the frequency is obtained for each luminance value of a predetermined width, and a histogram is created. When there are a plurality of faces, the face portions are sequentially selected to generate a histogram.

  Once the first face histogram is generated, it is next determined whether or not the frequency U at the luminance value zero is larger than the threshold value Uth (S3). Here, in order to determine whether the image is underexposed and has failed due to blackout, the frequency U and the threshold Uth are compared to determine whether the frequency U is greater than the threshold Uth. Therefore, the threshold value Uth may be a value that can be used to determine whether the image is a failed image due to insufficient exposure.

  If the result of determination in step S3 is that the frequency U is greater than the threshold value Uth, the first face portion of the subject is dark and blackened. In this case, it is next determined whether or not the frequency U is larger than the limit value Ulimit (S5). There is a relationship of limit value Ulimit> threshold value Uth, and it is determined whether or not an appropriate image can be obtained by correcting the brightness of the image.

  If the result of determination in step S5 is that the frequency U is not the limit value Ulimit, a blackout correction process is performed (S7). In this case, the threshold value Uth <frequency U <limit value Ulimit and the first face part of the subject can be said to be a failed image, but the so-called black color which corrects the brightness of the face part to a brightness appropriate for image processing. An appropriate image can be obtained by the correction processing for collapse.

  On the other hand, if the result of determination in step S5 is that the frequency U ≧ the limit value Ulimit, it is next determined whether or not there is a face image of the same person (S9). In this case, the threshold value Uth <the limit value Ulimit ≦ the frequency U, the subject is a failed image, and the image is extremely dark and has almost no shadow on the face. Can not do it. Therefore, image synthesis is performed in which the first face portion in the image is replaced with the face portion of the same person recorded in the face image DB 4b. In step S9, the face image DB 4b is searched, and it is determined whether or not the reference data for complementing the face portion of the same person is recorded in the face image DB 4b.

  If the result of determination in step S9 is that there is a face image of the same person, next, correction and composition processing for the first face is performed (S11). Here, in the example shown in FIG. 2, the face portion of the subject B22 is replaced with the complementary reference data of the face portion of the subject B22a of the same person. However, the brightness of the face of the subject B22a is appropriate exposure. If the face part is replaced as it is, the brightness of the face is greatly different from that of the surroundings, and an uncomfortable feeling is generated. Therefore, in the present embodiment, the brightness of the face is corrected so that it is brighter by a predetermined number of steps than the brightness of the background. When the brightness of the face based on the complementary reference data is corrected, image synthesis is performed to replace the first face of the original captured image. In addition to performing the image synthesis after correcting the face image, the face image is synthesized first, and after the synthesis, the image is corrected so that the exposure difference between the face image and the surrounding image is not noticeable. May be.

  If the result of determination in step S3 is that the frequency U is not greater than the threshold value Uth, it is next determined whether or not the frequency O at the maximum luminance value is greater than the threshold value Oth (S21). Here, in order to determine whether or not the first face is overexposed and a failed image, the frequency O is compared with the threshold value Oth, and it is determined whether the frequency O is greater than the threshold value Oth. . Accordingly, the threshold value Oth may be a value that can be used to determine whether or not the image is a failed image due to overexposure.

  If the result of determination in step S21 is that the frequency O is greater than the threshold value Oth, the first face of the subject is bright and overexposed. In this case, it is next determined whether or not the frequency O is larger than the limit value Olimit (S23). The limit value Olimit has a relationship of threshold value Oth <limit value Olimit. If the frequency O <limit value Olimit, an appropriate image can be obtained by correcting the brightness of the first face.

  If the result of determination in step S <b> 23 is that frequency O ≧ limit value Olimit is not satisfied, overexposure correction processing is next performed (S <b> 25). In this case, the threshold value Oth <the frequency O <the limit value Olimit, and the first face portion of the subject can be said to be a failed image. However, the so-called overexposure that corrects the brightness of the face portion to a brightness appropriate for image processing. An appropriate image can be obtained by the correction process.

  On the other hand, if the result of determination in step S23 is that frequency O ≧ limit value Olimit, next, as in step S9, it is determined whether there is a face image of the same person as the first face. (S27). In this case, the threshold value Oth <the limit value Olimit ≦ the frequency O, and the first face portion of the subject is a failed image, and the image is extremely white and has almost no shadow on the face. It cannot be a proper image. Therefore, image synthesis is performed in which the face portion in the image is replaced with the face of the same person recorded in the face image DB 4b. In this step S27, it is determined whether or not the complementary reference data for the face portion of the same person is recorded in the face image DB 4b.

  If the result of determination in step S27 is that there is a face image of the same person, then first face portion correction / combination processing is performed (S29). Here, the face portion that has become the failed image is replaced with the complementary reference data for the face portion of the subject B22a of the same person. As in the case of step S11, if the face part is replaced as it is, the background brightness of the face part is greatly different from the brightness of the face part, resulting in a sense of incongruity. The image correction may be made darker by a predetermined number of steps than the background brightness. When the brightness of the face based on the complementary reference data is corrected, image synthesis is performed to replace the first face of the original captured image. In addition to performing the image synthesis after correcting the face image, the face image is synthesized first, and after the synthesis, the image is corrected so that the exposure difference between the face image and the surrounding image is not noticeable. May be.

  When the processes in steps S7 and S11 are performed, or when the result of determination in step S9 is that there is no face image of the same person, or when the processes in steps S25 and S29 are performed, or the determination results in step S27 are the same. If there is no human face image, the face correction / combination processing is terminated. When a plurality of face parts are included in the photographed image, the second face part, the third face part, and the face part are sequentially selected, and the face correction / composition flow is executed. To do.

  As described above, in the first embodiment of the present invention, when the face portion is not properly exposed due to blackening or overexposure, an appropriate image is obtained by correcting the face portion. However, if the correct image cannot be obtained by the correction process (step S5 → Yes, S23 → Yes), the face portion is replaced using the complementary reference data recorded in the face image DB 4b. Image composition is performed (S11, S29). For this reason, it is possible to easily restore a captured image that has failed in a situation where the brightness difference is large, such as backlight. In particular, when the luminance difference is not so large, correction is performed using the captured image by the blackout correction process or the overexposure correction process. In addition, even when the luminance difference is large, the image composition is performed by replacing the face image, so that the failed image can be restored even under severe shooting conditions.

  In step S9 or S27, it is determined whether or not the face image of the same person is recorded in the face image DB 4b. However, the present invention is not limited to this, and the orientation of the face, the number of pixels of the face portion, exposure, focus state, etc. A routine for determining whether or not there is a face image suitable for image synthesis may be inserted. Further, when there are a plurality of face image candidates suitable for image synthesis, a routine for determining an optimum candidate may be inserted in consideration of the face direction, facial expression, brightness, focus, and the like.

  Next, a modification of the first embodiment of the present invention will be described with reference to FIGS. In the first embodiment, the degree of underexposure or overexposure is evaluated using the frequency U or the frequency O. On the other hand, in the present modification, the degree of black crushing is determined using a width ΔEu from a luminance value of zero to a predetermined frequency N or more as shown in FIG. 6A. I have to. Further, as shown in FIG. 6B, the degree of overexposure is determined using a width ΔEo of a predetermined frequency N or more from the maximum luminance value.

  The face correction / combination operation in this modification will be described with reference to the flowchart shown in FIG. In this flowchart, steps that perform the same processing as in the flowchart shown in FIG. 5 are given the same step numbers, and detailed descriptions thereof are omitted.

  In the flow shown in FIG. 7, first, a histogram of the first face is created (S1). Subsequently, it is determined whether or not the width ΔEu from the luminance value zero to the predetermined frequency N or more is larger than the threshold value ΔEuth (S4). Here, in order to determine whether or not the first face is underexposed and has become a failed image, the width ΔEu is compared with the threshold value Uuth to determine whether the width ΔEu is greater than the threshold value Uuth. . Therefore, the threshold value Uuth may be a value that can be used to determine whether or not it is a failed image due to blackout due to insufficient exposure.

  If the result of determination in step S4 is that the width ΔEu is smaller than the threshold Euth, it is next determined whether or not the width ΔEu is smaller than the limit value Eulimit (S6). It is determined whether or not an appropriate image can be obtained by correcting the brightness of the face portion because there is a relationship of threshold value Euth> limit value Eulimit. If the result of determination in step S6 is that the limit value Eulimit ≧ width ΔEu is not satisfied, a blackout correction process is performed (S7). The brightness of the face part is adjusted to an appropriate image by the blackout correction process.

  As a result of the determination in step S6, if the limit value Eulimit ≧ width ΔEu, as in the case of the first embodiment, it is next determined whether or not there is a face image of the same person (S9). In this case, since the first face portion of the subject is a failure image and there is almost no shadow of the face, an image in which the face portion of the photographed image is replaced with the face of the same person recorded in the face image DB 4b. Perform synthesis.

  If the result of determination in step S9 is that there is a face image of the same person, next, as in the first embodiment, correction / combination processing of the first face is performed (S11). Here, the face portion of the subject is replaced with the complementary reference data of the subject B22a of the same person. The image correction may be performed by a predetermined number of steps higher than the background brightness of the entire screen. In addition to performing the image synthesis after correcting the face image, the face image is synthesized first, and after the synthesis, the image is corrected so that the exposure difference between the face image and the surrounding image is not noticeable. May be.

  If the result of determination in step S4 is that the width ΔEo is not smaller than the threshold Eoth, it is next determined whether or not the width ΔEo greater than or equal to a predetermined frequency N from the maximum luminance value is smaller than the threshold Eoth ( S22). Here, in order to determine whether the image is overexposed and a failed image, the width ΔEo is compared with the threshold Eoth to determine whether the width ΔEo is smaller than the threshold Eoth. Therefore, the threshold Eoth may be a value that can be used to determine whether or not the image is a failed image due to overexposure.

  If the result of determination in step S22 is that the width ΔEo is smaller than the threshold value Eoth, the face portion is bright and the overexposed state. In this case, it is next determined whether or not the width ΔEo is smaller than the limit value Eolimit (S24). The limit value Eolimit has a relationship of limit value Eolimit <threshold value Eoth, and if limit value Eolimit <width ΔEoth <threshold value Eoth, an appropriate image can be obtained by correcting the brightness of the face portion of the image.

  If the result of determination in step S24 is that the limit value Eolimit is not greater than the width ΔEo, next, overexposure correction processing is performed as in the first embodiment (S25). In this case, limit value Eolimit <width ΔEoth <threshold value Eoth, and the first face portion of the subject can be said to be a failed image, but can be made an appropriate image by the overexposure correction process.

  If the result of determination in step S24 is that limit value Eolimit ≧ width ΔEo, it is next determined whether there is a face image of the same person as in the first embodiment (S27). Here, it is determined whether or not the complementary reference data for the face portion of the same person is recorded in the face image DB 4b.

  If the result of determination in step S27 is that there is a face image of the same person, next, as in the first embodiment, correction / combination processing of the first face is performed (S29). Here, the face portion that has become the failed image is replaced with the complementary reference data for the face portion of the subject B22a of the same person. As in the case of the first embodiment, if the face portion is replaced as it is, the image is corrected because it is greatly different from the surrounding brightness and uncomfortable. The image correction may be made darker by a predetermined number of steps than the background brightness of the entire screen. In addition to performing the image synthesis after correcting the face image, the face image is synthesized first, and after the synthesis, the image is corrected so that the exposure difference between the face image and the surrounding image is not noticeable. May be.

  When the processes in steps S7 and S11 are performed, or when the result of determination in step S9 is that there is no face image of the same person, or when the processes in steps S25 and S29 are performed, or the determination results in step S27 are the same. If there is no human face image, the face correction / combination processing is terminated. When a plurality of face parts are included in the photographed image, the second face part, the third face part, and the face part are sequentially selected, and the face correction / composition flow is executed. To do.

  As described above, in the modified example of the first embodiment of the present invention, the state of the blacked out or overexposed state of the face image portion is expressed by the width ΔEu from the luminance value zero to the predetermined frequency N or more, The determination is made using the width ΔEo of a predetermined frequency N or more from the maximum luminance value. Any index other than the frequency and width shown in the first embodiment and the modification may be applied to the embodiment of the present invention as long as it is an index that can determine a failed image due to brightness such as blackout or overexposure. be able to.

  In the first embodiment and the modification, the image data of the entire face is used when performing histogram analysis of the image data of the face portion. However, the present invention is not limited to this, and a part of the face may be selectively used. For example, the histogram analysis may be performed using image data of only a portion (for example, around the eyes) where it is desired to avoid the influence of blackout or overexposure except for the black hair portion.

  Next, a second embodiment of the present invention will be described with reference to FIGS. Similarly to the first embodiment, when the second embodiment becomes a failed image due to blackout or overexposure, the image is restored to an appropriate image by image processing. If the image cannot be restored, the appropriate image is restored by synthesizing the face image of the complementary reference data recorded in the face image DB 4b. In the first embodiment, the facial brightness is corrected in consideration of the ambient brightness when performing the facial composition processing in steps S11 and S29. In the second embodiment, the facial brightness is corrected. Describes a specific method for correcting the brightness of the face. The configuration in this embodiment is the same as the block diagram shown in FIG.

  Processing for restoring a captured image to an appropriate image in the present embodiment will be described with reference to FIGS. FIG. 8A shows a case in which the left side of the photographing screen is sufficiently illuminated, but the right side is not sufficiently illuminated due to backlight or strobe light distribution or emission amount restrictions. For this reason, an example is shown in which the subject A21 is exposed to sufficient light to achieve appropriate exposure, but the subject B22 is not exposed to sufficient light, resulting in a failed photo. If the face portion of the subject B22 cannot be seen well, in the present embodiment, as shown in FIG. 8B, the face image 22a of the subject B22 is searched for an image with a similar expression from the face image DB 4b, The image composition is cut out and the face portion is replaced. In this case, if the extracted face part is synthesized as it is, it does not match the brightness of the background and becomes unnatural. Therefore, the face image 22b is corrected to harmonize with the brightness of the background, and then image synthesis is performed. .

  The image composition shown in FIG. 8B will be described with reference to FIGS. FIG. 9A shows a luminance distribution obtained in a straight line shape of Y = Y1 in FIG. The face of the subject B22 is extremely dark with respect to the brightness of the background near the face of the subject A21, and the face cannot be seen well. Therefore, in the present embodiment, as shown in FIG. 9B, image synthesis is performed to replace the face image of the complementary reference data read from the face image DB 4b. In this case, the brightness of the face rather than the brightness of the background is about half of the brightness difference ΔEF between the average brightness of the face of the original subject B22 and the background brightness (ΔEF / 2), and the brightness of the background of the entire screen. Image processing is also performed. Since the brightness of the face is adjusted in consideration of the brightness of the background, the image synthesis is natural. In the present embodiment, the correction amount is about half (ΔEF / 2) of the luminance difference ΔEF between the average luminance of the face of the original subject B22 and the background luminance. In this way, by setting the ratio to about half, the background image and the face image can be harmonized. However, it is not limited to half and may be about one third or about two thirds.

  FIG. 10A shows a case where light is sufficiently applied to half of the face of the subject B22, but the other half is shaded. Such a situation can occur under strong sunlight or lighting. In such a case, it may be overlooked when viewing the image, assuming that there is no person in the shadow area. Therefore, in the present embodiment, as shown in FIG. 10B, the face portion of the subject B22 is searched for the face image 22a of the same person from the face images recorded in the face image DB 4b. Perform image composition to replace the face image. At this time, if the image synthesis is performed as it is, an uncomfortable image is obtained. Therefore, the facial image 22b is corrected so as to moderate the change in the luminance distribution of the facial image in consideration of ambient brightness. Perform synthesis.

  The image composition shown in FIG. 10B will be described with reference to FIGS. 11A and 11B. FIG. 11A shows a luminance distribution obtained in a straight line shape with Y = Y1 in FIG. The face portion of the subject B22 has a large brightness difference ΔEF, and the face cannot be seen well. Therefore, in the present embodiment, as shown in FIG. 11B, the brightness difference of the face image of the complementary reference data read from the face image DB 4b is corrected so as to be 1/2 of the brightness difference ΔEF. Then, image synthesis is performed to replace the corrected face image with the original face image.

  With the exposure amount at the time of shooting by the camera 10, the luminance difference of the face portion of the subject B22 becomes large. However, according to the present embodiment, since the luminance difference is halved, it is easy to observe the face of the subject B22. In addition, since the brightness of the face portion of the reference data for complementing read from the face image DB 4b is corrected so that the bright portion of the face substantially matches the brightness of the background, it matches with the surrounding brightness and has a natural feeling. It becomes an image.

  Next, face correction / combination operations in this embodiment will be described with reference to the flowchart shown in FIG. This signal processing is also executed by the signal processing and control unit 1 in accordance with a program stored in the flash memory 11. As in the first embodiment, the face correction / combination flow is performed when the camera 10 captures an image and records the image data in the recording unit 4 or when the recorded image data is read and displayed. To run.

  If the flow for face correction / combination is entered, first, a failed face portion is determined (S101). Here, the face detection unit 3 detects whether or not a face image is included. If a face image is detected as a result of this detection, the signal processing and control unit 1 determines whether the face image is a failed image that is not exposed to sufficient light and is not properly exposed. Determine whether. The detailed operation of the failed face determination will be described later with reference to FIG.

  Once the failed face portion determination has been carried out, it is next determined whether or not there is a failed face portion (S103). Here, the determination is made according to the result of the failed face determination performed in step S101. If the result of this determination is that there is a failed face, it is next determined whether or not correction is possible (S105). As described in the first embodiment, when the image has sufficient shadow, it can be restored to an appropriate image by performing image processing so that the image becomes bright (or dark) while maintaining the shadow. . In step S105, it is determined whether correction processing is possible.

  If the result of determination in step S105 is that correction is possible, correction processing is performed (S107). Here, the signal processing and control unit 1 changes the amplification factor of the image data according to the brightness level of the background light, and restores the appropriate image so that the face portion is appropriate.

  If correction is not possible as a result of the determination in step S105, it is next determined whether or not a face can be specified (S109). When the face portion cannot recognize shadows such as eyes and nose due to blackout or overexposure, it is difficult to search for a similar face. Therefore, in step S109, it is determined whether or not the feature part of the failed face can be extracted to the extent that a similar face can be searched from the face image DB 4b for the face determined as the failed face in steps S101 and S103.

  If the result of determination in step S109 is that face identification is possible, it is next determined whether or not there is a similar image in the face image DB 4b (S111). Here, a face image that matches or is similar to the failed face is searched from the face image DB 4b.

  If the face is not specified as a result of the determination in step S109, or if there is no similar face image as a result of the determination in step S111, then an uncorrectable display is performed (S113). Here, since it is impossible to restore an appropriate image by image synthesis by replacement of a face image, the fact is displayed on the display unit 8 or the like.

  If the result of determination in step S111 is that there is a similar face image, then a failure degree determination is performed (S115). The reason why the face part becomes a failure photo is that the face part is dark and fails, or there is a case where the face part is not shaded and fails. Since the restoration method to an appropriate image differs depending on the reason for failure, the degree of failure is determined in this step. Note that even if the face portion is dark, if there is sufficient shadow information, it can be restored by the correction processing in step S107. In step S115, it is determined whether or not the face portion is too dark to obtain sufficient shading information. Detailed operation of the failure degree determination will be described later with reference to FIG.

  If the failure degree determination is performed, the face image in the face image DB 4b is then corrected according to the failure degree (S117). Here, the face portion is cut out based on the complementary reference data recorded in the face image DB 4b, the brightness is adjusted according to the degree of failure, and the image composition for replacing the face portion is performed. Detailed operation for correcting the DB face image in accordance with the degree of failure will be described later with reference to FIG.

  After correcting the face image in the face image DB in step S117, next, image synthesis is performed by combining the size of the subject face image with the face image in the corrected face image DB and the position of the eyes and nose (S119). In step S117, the face image is corrected in accordance with the brightness of the background and the brightness difference of the face image. In this step, image synthesis for replacing the corrected face image with the face image in the original photographed image is performed. Do. In the image composition, the size of the face is matched with the position of the eyes and nose of the face.

  If there is no failed face as a result of the determination in step S103, or if correction processing in step S107 is performed, if uncorrectable display is performed in step S113, or if image synthesis is performed in step S119, face correction synthesis is performed. End the flow.

  Next, detailed operation of the failed face determination in step S101 will be described using the flowchart shown in FIG. In this flow, the face of the subject A21 in FIGS. 8 and 9 is referred to as a first face, and the face of the subject B22 is referred to as a second face.

  When the flow for determining the failed face portion is entered, in steps S131 to S149, it is determined whether or not the luminance difference between the background and the face portion is largely blackened. First, the average exposure EF1 of the first face is calculated (S131). Here, based on the image data output from the imaging unit 2, the average exposure EF1 of the image of the face portion of the subject A21 is obtained. Once the average exposure EF1 is obtained, next, the average exposure EF2 of the second face is calculated (S133). Here, based on the image data output from the imaging unit 2, the average exposure EF2 of the image of the face portion of the subject B22 is obtained.

  Once the average exposure EF2 of the second face is obtained, the average exposure EA of the background is then obtained (S135). Here, the average exposure EA of the background is calculated based on the image data of the area other than the face portions of the subject A21 and the subject B22 among the image data output from the imaging unit 2.

  Subsequently, it is determined whether or not a value obtained by subtracting the average exposure EF1 of the first face from the average exposure EA of the background is larger than a predetermined luminance difference ΔE1, that is, whether EA−EF1> ΔE1. (S137). Here, it is determined whether or not the first face is dark when viewed from the brightness of the background. Here, the luminance difference ΔE1 is a value for determining whether or not the photo is a failure due to blackout due to a large luminance difference between the background and the face portion.

  If the result of determination in step S137 is EA−EF1> ΔE1, the first face is determined to be dark. In this case, it is next determined whether or not the value obtained by subtracting the average exposure EF2 of the second face from the average exposure EA of the background is greater than a predetermined luminance difference ΔE1, that is, EA−EF2> ΔE1. It is determined whether or not (S139).

  As a result of the determination in step S139, if EA−EF2> ΔE1 is not satisfied, the first face is determined to be unsuccessful because the second face is not dark and the first face is dark (S143). On the other hand, if EA−EF2> ΔE1 as a result of the determination in step S139, both the first face and the second face are determined to have failed because both the first face and the second face are dark. (S141).

  As a result of the determination in step S137, if EA−EF1> ΔE1 is not satisfied, it is determined that the first face is not dark. In this case, next, as in step S139, it is determined whether or not EA−EF2> ΔE1 is satisfied (S145). As a result of this determination, if EA−EF2> ΔE1, the first face is not dark, but the second face is dark, so the second face is determined to be unsuccessful (S147). On the other hand, if EA−EF2> ΔE1 is not satisfied as a result of the determination in step S145, it is determined that there is no unsuccessful face because both the first face and the second face are not dark (S149).

  If the determination result of the failed face is stored in steps S141, S143, S147, and S149, next, the brightness determination of the face that is not failed is performed (S151). The determinations in steps S137 to S149 have been made based on the average exposure acquired in steps S131 to S135. However, if the brightness difference between light and dark in the face is large, even if the face is not a failure face when judged from the average exposure, the face may fail as described with reference to FIG. . That is, the bright part may be overexposed, and conversely the dark part may be underexposed. Therefore, in step S151, the brightness difference in the face is determined.

  If a luminance difference is detected in step S151, it is determined that the face is a failed face if the luminance difference is equal to or greater than ΔE2 (S153). The luminance difference ΔE2 may be appropriately selected as a design value, but may be, for example, about two to three stages. If there is such a luminance difference, it becomes difficult to identify shadows of areas that are too bright or too dark. Further, there is a possibility that the bright part is overexposure or the dark part is underexposure. In this step, although not shown in detail, it is determined whether or not the luminance difference is greater than or equal to ΔE2 for each of the first and second faces, and the result is temporarily stored. If the failure face is determined in step S153, the process returns to the original flow.

  As described above, in the failure face determination flow, it is determined whether the first and second faces have a large difference in average exposure, and the failure faces are determined to be failure faces. It is determined whether or not the face is a failure due to a large brightness difference. In this flow, the case of determining only the first and second faces has been shown, but if there are three or more faces, the same determination may be made. Further, in steps S131 to S149, it is determined that the shadow is blacked out and the shadow information of the face is lost. However, it is also determined that the shadow information of the face is lost due to overexposure. You may make it do.

  Next, detailed operation of the failure degree determination in step S115 will be described using the flowchart shown in FIG. When the failure degree determination flow is entered, it is first determined whether or not the first face is dark and failed (S161). In Steps S143 and S141, when the first face is dark enough that the shadow information is not sufficiently obtained, the fact that the face is a failed face is temporarily stored, and therefore determination is made based on the storage result.

  If the result of determination in step S161 is that the first face is dark and fails, then the difference ΔEF1 between the first face and the background is obtained (S165). Here, a difference between the average exposure EA of the background obtained in step S135 and the average exposure EF1 of the first face obtained in step S131, that is, EA−EF1 = ΔEF1 is calculated.

  If the result of determination in step S161 is that the first face is dark and has not failed, it is next determined whether or not the first face has failed due to shading (S163). In steps S151 and S153, if there is a shadow on the first face, the brightness is ΔE2 or more, and if there is a possibility of blackout or overexposure, it is determined as a failed face, and this is temporarily stored. Then, it determines based on the result of temporary storage.

  If the result of determination in step S163 is that the first face has failed due to a shadow, the distribution of the shadow within the face of the first face is determined (S167). In this step, as described with reference to FIG. 11A, the difference ΔEF between the highest luminance and the lowest luminance in the face is calculated based on the image data output from the imaging unit 2.

  If the result of determination in step S163 is that there was no failure due to the shadow of the first face, or if the determination in step S165 or S167 was made, it is next determined whether or not the second face was dark and failed ( S171). In Steps S147 and S141, if the second face is dark enough that the shadow information is not sufficiently obtained, the fact that the face is a failed face is temporarily stored. Therefore, the determination is based on the storage result.

  If the result of determination in step S171 is that the second face is dark and fails, then the difference ΔEF2 between the second face and the background is obtained (S175). Here, the difference between the average exposure EA of the background obtained in step S135 and the average exposure EF2 of the second face obtained in step S133, that is, EA−EF2 = ΔEF2 is calculated.

  If the result of determination in step S171 is that the second face is dark and has not failed, it is next determined whether or not the second face has failed due to shading (S173). In steps S151 and S153, if there is a shadow on the second face, the brightness is equal to or greater than ΔE2, and there is a possibility of blackout or overexposure, it is determined as a failed face, and this is temporarily stored. Then, it determines based on the result of temporary storage.

  If the result of determination in step S173 is that the second face has failed due to a shadow, the distribution of the shadow within the face of the second face is determined (S177). In this step, as described with reference to FIG. 11A, the difference ΔEF between the highest luminance and the lowest luminance in the face is calculated based on the image data output from the imaging unit 2.

  As a result of the determination in step S173, if the failure is not due to the shadow of the second face, or if the determination in step S175 or S177 is performed, the process returns to the original flow.

  As described above, in the failure degree determination flow, the luminance differences ΔEF1 and ΔEF2 from the background are determined for each of the first and second faces, or the luminance difference ΔEF in the face is determined for each of the first and second faces. Judgment and seeking the degree of failure. In this flow, the case of determining only the first and second faces has been shown, but if there are three or more faces, the same determination may be made. In steps S161 and S171, it was determined whether the face was dark and blacked out and failed. However, when the face is bright and overexposed and the shadow information of the face is lost, the brightness of the background The difference may be determined.

  Next, detailed operation for correcting the DB face image in accordance with the degree of failure in step S117 will be described using the flowchart shown in FIG. When the DB face image correction flow is entered in accordance with the degree of failure, first, as in step S161, it is determined whether or not the first face is dark and failed (S181).

  If the result of determination in step S181 is that the first face is dark and fails, then a composition image is created with the exposure amount of the DB image of the first face as background minus ΔEF1 / 2 (S185). Here, for the face image similar to the failed face retrieved from the face image DB 4b in step S111, half the difference ΔEF1 from the first face background calculated in step S165 (ie, 1 / 2), a face image with reduced brightness is generated as a composition image (see FIG. 9B).

  If the result of determination in step S181 is that the first face is dark and has not failed, it is next determined whether or not the first face has failed due to shading as in step S163 (S183). .

  If the result of determination in step S183 is that the first face has failed due to a shadow, then the exposure amount of the first face DB image is matched with the first face in the screen, and the in-face shadow is determined. An image for synthesis is created with a distribution of 1/2 (S187). Here, in step S111, the in-face shadow of the face image similar to the failed face retrieved from the face image DB 4b is ½ of the luminance difference ΔEF in the first face calculated in step S167. Is generated (see FIG. 11B).

  If the result of determination in step S183 is that there was no failure due to the shadow of the first face, or if an image for synthesis was created in step S185 or S187, then the second face was dark and failed as in step S171. It is determined whether or not (S191).

  If the result of determination in step S191 is that the second face is dark and fails, then a composition image is created with the exposure amount of the DB image of the second face set to the background minus ΔEF2 / 2 (S195). Here, for the face image similar to the failed face retrieved from the face image DB 4b in step S111, half the difference ΔEF2 from the first face background calculated in step S165 (ie, 1 / 2), a face image with reduced brightness is generated as a composition image (see FIG. 9B).

  If the result of determination in step S191 is that the second face is dark and has not failed, it is next determined whether or not the second face has failed due to shading as in step S173 (S193). .

  If the result of determination in step S193 is that the second face has failed due to a shadow, then the exposure amount of the DB image of the second face is matched with the second face in the screen, and the shadow within the face is matched. An image for synthesis is created with a distribution of 1/2 (S197). Here, in step S111, the in-face shadow of the face image similar to the failed face retrieved from the face image DB 4b is set to 1/2 of the brightness difference ΔEF in the second face calculated in step S177. Is generated (see FIG. 11B).

  If the result of determination in step S193 is that the shadow of the second face was not a failure, or if an image for synthesis has been created in step S195 or S197, the original flow is returned to. When returning to the original flow, as described above, the face image combined in this flow and the original subject image are combined so that the eye-nose positions coincide (see S119).

  Thus, in the flow of correcting the DB face image in accordance with the failure degree, if the face portion is darker than the background and the shadow information is not sufficiently obtained, the failure image is recorded and recorded in the face image DB 4b. An image for image composition to be replaced with the face portion of the same person is generated. In this case, if the face image recorded in the face image DB 4b is used as it is, if the brightness difference from the background is large, the image becomes uncomfortable, and in this embodiment, the brightness of the background and the face. The difference is corrected to be half that of actual photographing (see S185 and S195). In this embodiment, the correction amount is ½ of the luminance differences ΔEF1 and ΔEF2, but this value may be appropriately changed as a design matter. In addition to the constant multiple of the luminance difference, it is of course possible to have a constant luminance difference.

  In the present embodiment, when the shadow in the face largely fails, an image in which the shadow in the face is compressed by half is generated as a synthesis image (see S187 and S197). This makes it possible to correct the face image with a natural-looking shadow. In this embodiment, the correction amount is ½ of the luminance difference ΔEV. However, the correction amount is not limited to this, and may be appropriately changed as a design matter.

  In the flow shown in FIG. 15, the case where only the first and second faces are determined is shown, but if there are three or more faces, the determination may be made in the same manner. If the face is too bright and overlooked, and the face has failed, the face image retrieved from the face image DB 4b is brightened and corrected by half the brightness difference between the face portion and the background. The corrected face image may be used as a composition image.

  Next, a modified example of the image composition performed in step S119 will be described with reference to FIGS. In the first and second embodiments of the present invention, the image composition replaced with the face recorded in the face image DB 4b is simply displayed as a still image at the time of composition. In the modification of the second embodiment, when image synthesis is performed, moving image expression is performed when a face image is replaced.

  Now, assume that image composition is performed in which the face portion of the subject B22 shown in FIG. 10A is replaced with the face image 22a recorded in the face image DB 4b. In performing this image synthesis, first, the face image of the subject B22 to be replaced is enlarged and displayed so as to zoom in from the size shown in FIG. 10A to the size shown in FIG. This enlarged display is performed while changing from a small image to a large image by an animation effect.

  Then, when the enlarged display is performed as shown in FIG. 10B, image composition is performed in which the face image 22a of the same person is replaced from the supplementary reference data recorded in the face image DB 4b. Also in this case, as described with reference to FIGS. 10 and 11, it is preferable to combine the images after correcting the brightness of the face image 22 b so that the brightness of the face is considered and the change in brightness of the face is reduced.

  Next, the flow for performing the moving image expression described with reference to FIG. 16 will be described with reference to the flowchart shown in FIG. When the flow of synthesis of the subject face image and the corrected DB face image is entered, it is first determined whether or not to perform moving image expression (S201). In the case of performing moving image expression, in this modification, since the setting is made in advance on the menu screen or the like, it is determined in this step whether or not the moving image expression is set on the menu screen or the like.

  If the result of determination in step S201 is to express moving images, then the ratio R1 between the screen size and the subject face size, ie, R1 = (screen size) / (subject face size). ) Is obtained (S203). Here, for example, in the example shown in FIG. 16A, the size of the screen is divided by the size of the face of the subject B22, and this calculation result is R1.

  If the screen-to-face size ratio R1 is obtained in step S203, then the face size of the subject is gradually multiplied by R1 (S205). Here, as described with reference to FIG. 16, the size of the face of the subject B22 is gradually increased by the animation effect. While gradually expanding and displaying the face, it is determined whether or not the face has reached R1 (S207). If the result of this determination is that R1 has not been reached, processing returns to step S205 and enlargement display continues. When R1 is reached, the size of the face of the subject B22 becomes approximately the size of the screen.

  As a result of the determination in step S207, when the R1 times is reached, the DB face image is made the same size, and the eyes and nose positions are aligned and fitted into the image (S209). Since the size of the face image has reached R1 times, the face image is approximately the size of the screen. Therefore, image synthesis is performed in which the face image having a substantially screen size is replaced with the face image for synthesis generated in step S117. At this time, alignment is performed so that the eye-nose position of the original face image and the eye-nose position of the face image for synthesis are substantially matched, and then the image is replaced. When the image replacement is completed, the DB face image is restored to the original size.

  If the result of determination in step S <b> 201 is that it is not a moving image expression, the image composition for facial image replacement is performed without performing the moving image expression, as in the second embodiment of the present invention. For this purpose, first, a ratio R2 between the face size of the subject and the face image size of the face image DB, that is, R2 = (face size of subject) / (size of DB face image) is obtained (S211). ). Here, for example, in the example shown in FIG. 16, division is performed by the size of the face of the subject B22 shown in FIG. 16 (a) and the size of the face image 22a shown in FIG. 16 (c). And

  Once the ratio R2 is obtained, the DB face image is then multiplied by R2 and fitted into the image (S213). Here, for example, the face size of the same person retrieved from the face image DB 4b is corrected so as to be the same size as the face image of the subject of the photographed image, and then image composition for replacing the face image is performed. When the image replacement in step S209 or S213 is performed, the process returns to the original flow.

  As described above, in the modification of the second embodiment of the present invention, when the image composition for replacing the face image of the same person retrieved from the face image DB 4b is performed, the face image to be replaced is temporarily displayed in an enlarged manner. However, the image is replaced when the image is enlarged. If only the face part is replaced with the face image retrieved from the face image DB 4b, there is a possibility that an unnatural photograph will be obtained if the clothes are dark but only the face is bright. As in this modification, when the face is enlarged and the screen is full, replacing the image retrieved from the face image DB 4b (S205 → S209) enables more natural display.

  Next, creation of a face image database in the face image DB 4b according to the second embodiment of the present invention will be described with reference to FIGS. First, generation of a face image to be recorded in the face image database will be described. A face image 23 shown in FIG. 18 shows a face portion in a photographed image photographed by the camera 10. In recording in the face image database, first, the face size of the face image 23 is normalized. That is, the size is changed to a normalized face image 23a having a certain size. Further, when recording in the face image database, the face image is rotated so that the nose muscle line 27 is in the vertical direction.

  When the nose line 27 is set in the vertical direction, an image as shown in FIG. Further, as shown in FIG. 19 (c), when there is another face image 25 with the nose line 27 vertical as shown in FIG. 19 (c), from the face image 23 and the face image 25, as shown in FIG. 19 (b). In addition, a face image 24 as shown in FIG. 19B is generated by morphing. As a result, three types of face images 23 to 25 can be recorded in the face image DB 4b.

  FIG. 20 shows an example of a face image database recorded in the face image DB 4b created by using the face image generation described with reference to FIGS. In this face image database, for each of the individual A, B,..., The nasal lines 27 are faced in different directions from the center of the face, 0 degrees, 10 degrees to the left, 20 degrees to the left,. Recorded in a matrix. The expression 1 is a smile, the expression 2 is an ordinary face, and the expression 3 is an angry face. In this example, the orientation of the face is every 10 degrees, and there are three types of facial expressions. However, the present invention is not limited to this, and the face image is recorded by classifying less or more than this. But of course it does n’t matter. Further, although one intermediate image is generated by morphing, the number is not limited to one, and a plurality of intermediate images may be generated.

  Next, a flow for creating the above-described face image database will be described using the flowchart shown in FIG. If the flow for creating the face image database is entered, first, personal authentication determination is performed (S301). Here, an individual is specified using the facial features detected by the face detector 3.

  Once personal authentication determination is made, facial expression / angle determination is then performed (S303). Here, facial expressions such as a smile, a normal face, and an angry face are determined using the detection result of the eyes and mouth by the face detection unit 3. Further, the angle of the face orientation is also determined using the nose line 27 detected by the face detection unit 3.

  Once facial expression / angle determination is performed, it is next determined whether authentication is impossible (S305). Here, it is determined whether or not an individual has been identified as a result of the personal authentication determination in step S301. If the result of this determination is that personal authentication has failed, manual tagging is performed (S307). Since the face image cannot be personally authenticated, the user manually attaches a tag such as a personal name as appropriate.

  Once manual tagging has been performed, it is next determined whether or not there is another person who is the same as the person to whom the manual tag is attached (S309). Here, it is determined whether or not the same person exists in the face image to which the manual tag is attached. As a result of this determination, if there is no same person, it is classified as a new individual (S311).

  If the result of determination in step S305 is that personal authentication has been made, it is next determined whether or not face images having the same facial expression and angle are recorded in the face image DB 4b (S307). As described with reference to FIG. 20, in the face image database, a database is created based on the angle and expression of the face, and if already recorded, no further recording is performed. In step, it is determined whether or not recording has been completed.

  As a result of determination in step S307, when face images of the same expression / angle are not recorded, or as a result of determination in step S309, there is the same person, or when classification as a new individual in step S311 Next, a face image is cut out to be a face image (S313). Here, the face portion detected by the face detection unit 3 is trimmed by the signal processing and control unit 1.

  Once the face image is generated, the face size is normalized to 1/5 of the screen, and the face image is rotated so that the nose is in the vertical direction (S315). Here, as described with reference to FIG. 18, the size of the face and the direction of the nose are aligned. The size of the face is an example, and it may be larger or smaller than 1/5 of the screen, but about 1/5 is a size that is easy to handle in consideration of the size of the face in a normal captured image. .

  Subsequently, the face image corrected in step 315 is registered as an individual face-to-face image (S317). Here, a face image corresponding to the personal name is recorded in the face image DB 4b as shown in FIG.

  Once the face image is registered, it is next determined whether morphing is possible (S319). Morphing is image processing for generating an intermediate image using two images as described with reference to FIG. In this step, it is determined whether or not there are two face images as shown in FIGS.

  If the result of determination in step S319 is that morphing is possible, an angle intermediate image is generated (S321). Here, for example, as shown in FIG. 19, an intermediate face image 24 is generated from the face image 23 and the face image 25 by morphing. Once the angle intermediate image is generated, it is registered as an individual face-to-face image as in step S317 (S323). Here, the angle intermediate image generated in step S321 is recorded in the face image DB 4b in association with the personal name as shown in FIG.

  When the face image is registered in step S323, or when the morphing is impossible as a result of the determination in step S319, or when the face image of the same expression / angle is already registered as a result of the determination in step S307, The flow for creating the face image database is terminated, and the process returns to the main routine.

  As described above, in the second embodiment of the present invention, when the failed image cannot be made an appropriate image by the correction process (step S105 → No), the supplementary image recorded in the face image DB 4b is used. By using the reference data, image composition for replacing the face portion is performed. For this reason, it is possible to easily restore a captured image that has failed in a situation where the brightness difference is large, such as backlight.

  Further, in the present embodiment, when replacing the face part, the reason for the failure is read from the face image DB 4b depending on whether the face part is entirely dark or the difference in brightness of the face part is large. The correction of the face image complementary reference data is different. For this reason, it is possible to perform appropriate image composition according to the failure.

  Furthermore, in this embodiment, when correcting the complementary reference data of the face image read from the face image DB 4b, the brightness of the face is corrected in consideration of the surrounding brightness. For this reason, the restored photograph can be made natural.

  Further, in the present embodiment, when the face image is replaced with the complementary reference data of the face image read from the face image DB 4b, the size of the face image and the position of the eyes and nose are combined and combined. For this reason, the face of the restored image matches the surroundings and feels natural. Although it is desirable that both the size and the eyes and nose are aligned, only one of them may be used.

  Further, in the present embodiment, as shown in FIGS. 18 to 21, a face image for a face image database is generated based on a photographed image acquired by photographing. For this reason, the photographer can generate a face image database through normal photographing. In addition, even the same person is recorded in the database for each facial expression and face angle, so that an optimal face image can be selected according to the shooting situation. When various face images are registered in the face image database by the method shown in FIGS. 18 to 21, when searching for a similar face image in step S111, it is optimal according to the face direction and facial expression. It is possible to select a correct face image.

  In this embodiment, the determination of whether or not the face is a failed face in steps S101 and S103 is based on the difference between the luminance of the face and the average luminance of the background, or the luminance difference in the face. Of course, the determination may be made by other methods such as the method described in the first embodiment.

  Next, a third embodiment of the present invention will be described with reference to FIG. In the first and second embodiments, the present invention is applied to a camera having an imaging unit. However, in the present embodiment, an image captured by a camera or the like is recorded in a recording device such as a personal computer (PC). This is an example in which the present invention is applied to the case where the image is recorded, when the recorded photographed image is read and reproduced and displayed.

  FIG. 21 is a block diagram showing a configuration of the PC 50 according to the present embodiment. The PC 50 includes a microcomputer 61, an operation input device 71, a display monitor 72, an external image input device 73, and a recording device 74. The microcomputer 61 has hardware such as a storage unit for recording programs and data, a control unit, and the like, and each means for performing functions by these hardware. Each means includes a face determination means 62, a histogram generation means 63, an appropriateness determination means 64, a correction means 65, and a composition means 66.

  Based on the image data of the captured image input from the external image input device 73, the face determination means 62 has a face in the captured image, and if it exists, its size, position, feature, etc. Determine. The histogram generation means 63 generates a histogram as shown in FIGS. 3, 4, 6, etc. based on the image data.

  Based on the histogram generated by the histogram generation unit 63, the appropriateness determination unit 64 determines whether the image is a proper image, in other words, a failure image. The correction unit 65 corrects the brightness when the image is determined to be a failure image by the appropriateness determination unit 64 and corrects the brightness and records the face portion of the photographed image. When replacing with a face image recorded in the device 74, the brightness of the face image is corrected in consideration of the brightness of the background.

  The synthesizing unit 66 searches for a matching face from the face images recorded in the recording device 74 when the photographed image is a failed image and does not become an appropriate image due to brightness correction, and Image synthesis is performed in which the face portion is replaced with the searched face image.

  An operation input device 71, a display monitor 72, an external image input device 73, and a recording device 74 are connected to the microcomputer 61, and data and control commands can be communicated. The operation input device is an input instruction device such as a keyboard or a mouse. The display monitor 72 is a display device such as a liquid crystal monitor, and displays a captured image input by the external image input device 73 and a captured image recorded in the recording device 74.

  The external image input device 73 is a device for inputting an image taken by the camera 10 or the like to the PC 50, and is an input device of an external memory such as a USB memory or an input device through a communication line such as the Internet.

  The recording device 74 is an internal or external memory in the PC 50, and records a photographed image input by the external image input device 73, a face image database created using a face image from the photographed image, and the like. . In the case where the face image database is not provided, a face portion may be cut out from the photographed image at the time of image composition and this face image may be used.

  The PC 50 according to the present embodiment executes the flowchart shown in FIG. 5 or FIG. 7 according to the first embodiment when the image data of the photographed image is input from the outside by the external image input device 73, and corrects the photographed image. After performing the synthesizing process, image data is recorded in the recording device 74. Alternatively, the image data of the photographed image input from the outside is recorded in the recording device 74 as it is, and when the recorded image data is read and reproduced, the flowchart shown in FIG. 5 or 7 is executed to correct the photographed image. Display the combined image on the display monitor 72.

  As described above, the PC 50 according to the third embodiment of the present invention performs the same image correction and correction as in the first embodiment when recording the captured image on the recording device 74 or when reproducing and displaying the recorded captured image. I'm trying to compose. For this reason, even when the face portion of the photographed image becomes a failed image due to blackout or whiteout, it can be restored to an appropriate image.

  In the present embodiment, each means is provided in the microcomputer 62 so as to perform the same function as in the first embodiment. However, the present invention is not limited to this, and the same function as in the second embodiment is performed. Of course, each means may be provided.

  In the present embodiment, the PC 50 is provided with the microcomputer 61 and other devices. However, it is of course possible to provide some devices, for example, the recording device 74 in an external server. .

  As described above, in each embodiment of the present invention and its modifications, it is possible to correct the brightness of a photographed image that has failed due to blackout or overexposure in a situation where the luminance difference is large such as backlight. If this is the case, this correction is performed to obtain an appropriate image. Also, if the image cannot be made appropriate by correcting the brightness of the image due to extreme blackout or excessive whiteout, the image composition is replaced with the face of the same person recorded in the database. The proper image is obtained by performing For this reason, it is possible to easily restore a failed photographed image.

  In addition, when synthesizing an image to be replaced with the face of the same person recorded in the database, the brightness of the face is corrected in consideration of the brightness of the background, and a face image for synthesis is generated. Yes. For this reason, it is possible to generate a composite image that does not have a sense of incongruity with ambient brightness.

  In each embodiment and modification of the present invention, image synthesis is performed to replace the face portion of the photographed image after performing brightness correction on the face image stored in the database. However, the present invention is not limited to this, and the brightness of the face portion may be corrected after the face portion of the photographed image is replaced with the face image stored in the database. In addition to correcting the brightness of the face part, another method is to correct the brightness of the subject around the face part so that the face part becomes inconspicuous as a whole. You may make it give.

  In each embodiment and modification of the present invention, a face database (for example, see FIG. 20) is generated in advance and stored in the recording unit 4 (face image DB 4b). However, as the face part database, the photographed image itself recorded in the recording unit 4 is used, and when the face part image is synthesized, the face part is cut out from the recorded photographed image. Of course, it does not matter if image synthesis is performed.

  Furthermore, in each embodiment and modification of the present invention, correction and composition are performed so that a human face is an appropriate image, but the same applies to animals such as pets. be able to. In this case, data for identifying the face of an animal such as a pet may be used.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone, a personal digital assistant (PDA), a game device, or the like.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Signal processing and control part, 1b ... Poor exposure determination part, 2 ... Imaging part, 3 ... Face detection part, 4 ... Recording part, 4b ... Face image DB, 5 ... Strobe unit, 6 ... Operation determination unit, 7 ... Audio recording unit, 8 ... Display unit, 9 ... Clock unit, 10 ... Camera, 11 ... Flash memory, 12 ... clock part, 21 ... subject A, 22 ... subject B, 22a ... face image, 22b ... face image, 23 ... face image, 23a ... normalized face image, 24 ... Face image, 25 ... Face image, 27 ... Nose lines, 50 ... Personal computer (PC), 61 ... Microcomputer, 62 ... Face determination means, 63 ..・ Histogram generation means, 64... Appropriateness determination means, 65... Correction means, 66. Work input device, 72 ... display monitor, 73 ... external image input device, 74 ... recording device

Claims (11)

A face image database for storing an image including a face portion;
An appropriate image determination unit for determining whether the image is appropriate based on the brightness of the face portion in the captured image;
A first correction unit that corrects the captured image when the image is determined to be a failed image by the appropriate image determination unit and becomes an appropriate image by correcting the brightness of the captured image;
When it is determined that the image is a failed image by the appropriate image determination unit, if the image cannot be made an appropriate image by correcting the brightness of the captured image, it matches the face in the failed image from the face image database. Alternatively, a second correction unit that searches for a similar face image and corrects the brightness of the searched face image;
An image composition unit for performing image composition for replacing the face image corrected by the second correction unit with the face image of the failed image;
A face image synthesizing device characterized by comprising: The face image synthesizer according to claim 1 , wherein the appropriate image determination unit generates a histogram of the captured image and makes a determination based on the histogram. The face image composition device according to claim 1 , wherein the appropriate image determination unit makes a determination based on a difference between an average brightness of a face portion of the photographed image and a brightness of a background. The face image synthesizing apparatus according to claim 1 , wherein the appropriate image determining unit determines based on a difference in brightness in a face portion of the photographed image. The second correction unit has a brightness obtained by subtracting a correction amount based on a difference between an average brightness of a face portion of the photographed image and a brightness of the background from the brightness of the background. face image synthesizing apparatus according to claim 1, characterized in that the corrected so that. If the second correction unit determines that the image is a failure image based on the difference in brightness in the face portion of the photographed image, the second correction unit applies the brightness and darkness in the face portion to the searched face image. The face image composition apparatus according to claim 1 , wherein the difference is corrected so as to be compressed. 2. The face according to claim 1 , wherein the synthesis unit replaces the image so that the eye-nose position of the face image corrected by the second correction unit matches the eye-nose position of the face image of the failure image. Image composition device. The face image synthesizing apparatus according to claim 1 , wherein the database stores face images having different expressions and face orientations for each person. And a display unit for displaying an image;
A display control unit for an image to be displayed on the display unit;
Have
The display control unit enlarges and displays the face portion of the photographed image, enlarges and displays the face image synthesized by the synthesis unit, and shoots including the synthesized face image. The face image synthesizing apparatus according to claim 1 , wherein the face image synthesizing apparatus displays the reduced image to the original size. Based on the brightness of the face part in the captured image, determine whether it is an appropriate image,
When it is determined as a failed image by determining whether or not the image is appropriate, if the image becomes a proper image by correcting the brightness of the captured image, the captured image is corrected,
If it is determined that the image is a failure image by determining whether the image is appropriate and cannot be made an appropriate image by correcting the brightness of the captured image, the face image database that stores an image including a face portion is used. Search for a face image that matches the face in the failed image, correct the brightness of the searched face image,
Performing image synthesis to replace the face image corrected by the correction with the face image of the failed image;
A face image synthesis method characterized by the above. In a computer program for performing face image synthesis in a computer having a face image database,
Determining whether the image is appropriate based on the brightness of the face portion in the captured image;
A step of correcting the photographed image when it is determined that the image is a failure image by determining whether or not the image is a proper image, and a correct image is obtained by correcting brightness of the photographed image;
If the image is determined to be a failed image by determining whether or not the image is appropriate and cannot be made the correct image by correcting the brightness of the captured image, it matches the face in the failed image from the face image database. Searching for a face image to be corrected, correcting the brightness of the searched face image,
Performing image synthesis for replacing the face image corrected by the correction with the face image of the failed image;
A program that causes a computer to execute.