JP2012003324A - Image processing system, imaging apparatus, image processing program and memory medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily create images on which a subject is good looking and gives no unnatural impression.SOLUTION: An imaging processing system includes a image input portion for inputting a first image, a face detection part for detecting face regions of the first image input by the image input portion, a image correction portion for performing correction processing with respect to the face region of the first image based on a second image different from the first image and having character amount which is same as the character amount of the face regions of the first image detected by the face detection portion.

Description

  The present invention relates to an image processing device, an imaging device, an image processing program, and a storage medium.

  Conventionally, a technique for correcting an image obtained when a person is photographed and generating an image in which the person in the image looks beautiful has been used. As a technique for generating an image that makes a person in such an image look beautiful, for example, correction for enlarging the pupil of the person in the image and correction for smoothing the skin can be cited (see Patent Document 1).

JP 2005-159561 A

  However, in the technique of Patent Document 1, since correction is performed at a uniform rate regardless of the state of the person in the image and the shooting environment, the natural texture such as the face color and facial expression of the person in the image is impaired, which is unnatural. It is easy to become an image.

  Therefore, an object of the image processing apparatus, digital camera, image processing program, and storage medium of the present invention is to easily generate an image that makes a subject look beautiful and does not give an unnatural impression.

  The image processing apparatus of the present invention is different from the first image, an image input unit that inputs a first image, a face detection unit that detects a face area of the first image input by the image input unit, An image correction unit that performs correction processing on the face area of the first image based on the second image having the same feature quantity as the feature quantity of the face area of the first image detected by the face detection unit. With.

  In addition, when the first image is corrected by the image correction unit, a display unit that displays one or a plurality of the second images and the second image displayed on the display unit serve as a reference for the correction process. You may provide the selection part which selects a reference | standard image.

  The image processing apparatus may further include an image storage unit that stores one or more second images, and a selection unit that selects a reference image serving as a reference for the correction process from the second images stored in the storage unit. Good.

  Further, the second image may be stored in the image storage unit together for each subject included in the second image.

  The second image may be an image having at least a part of a face area.

  The first image corrected by the image correction unit may be stored in association with correction information indicating the content of the correction process.

  The first image associated with the correction information may be stored in the image storage unit as the second image.

  The correction process may include at least one of color correction, contrast correction, brightness correction, or correction for a facial feature.

  An imaging apparatus according to the present invention includes the above-described image processing apparatus.

  The image processing program of the present invention is different from the image input step of inputting a first image, the face detection step of detecting the face area of the first image input by the image input step, and the first image, An image correction step of performing correction processing on the face area of the first image based on the second image having the same feature quantity as the feature quantity of the face area of the first image detected by the face detection unit. Can be executed on a computer.

  The recording medium of the present invention is a computer-readable recording medium in which the image processing program is stored.

  According to the image processing device, the imaging device, the image processing program, and the storage medium of the present invention, it is possible to easily generate an image that makes a subject look beautiful and does not give an unnatural impression.

It is a block diagram which shows the structure of the imaging device 10 in this embodiment. It is a flowchart which shows the operation | movement at the time of imaging | photography. It is a display example of a list of thumbnail images corresponding to sample images.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 10 according to the present embodiment. The imaging device 10 performs skin beautification processing on a person in an image in addition to the image processing in addition to the image capturing mode in which image processing is performed on an image acquired by image capturing based on image capturing conditions set in advance. Has a beautiful skin shooting mode. This skin beautifying mode is composed of a first skin beautifying mode for performing beautifying processing based on a reference image, which will be described later, and a second skin beautifying mode for performing beautifying processing based on user settings. In the present embodiment, a case where a person is photographed as a subject will be described.

  As shown in FIG. 1, the imaging device 10 includes an imaging lens 11, an imaging element 12, an A / D conversion unit 13, a buffer memory 14, an image processing unit 15, a display unit 16, an operation unit 17, a memory 18, a recording I / O. An F unit 19, a recording medium 20, a control unit 21, and a bus 22 are provided.

  The imaging lens 11 forms a subject image on the imaging surface of the imaging element 12. The imaging element 12 photoelectrically converts the subject light that has passed through the imaging lens 11 and outputs analog image signals corresponding to R, G, and B colors. The imaging device 12 performs thinning-out reading at predetermined intervals during imaging standby, and outputs an image signal that is a source of a through image. Thereafter, the image sensor 12 outputs an image signal that is the basis of the main image when a release button 23 described later is fully pressed.

  An image signal output from the image sensor 12 is input to the A / D converter 13. The A / D converter 13 converts the analog image signal output from the image sensor 12 into a digital image signal. The digital image signals are collected into one frame and recorded in the buffer memory 14 as image data. The buffer memory 14 temporarily records image data in the pre-process and post-process of image processing by the image processing unit 15 described later.

  The image processing unit 15 performs image processing on the image data recorded in the buffer memory 14. Examples of the image processing include well-known white balance adjustment, color interpolation, gradation conversion processing, and contour enhancement processing.

  In addition, the image processing unit 15 performs a skin beautification process on the face area detected from the main image. Examples of skin beautifying processes include exposure correction to correct overexposure, soft focus processing, saturation correction, lightness correction, correction to make the cheek look narrower, adjustment of characteristic parts (eyebrows, eyes, mouth, etc.) in the face area, Examples include a process for suppressing highlights. For example, correction for making the cheek look narrow is performed by correcting the contour of the face as disclosed in Japanese Patent Application Laid-Open No. 2004-264893. Further, correction of the characteristic part in the face area includes correction of enlargement of the eye area and narrowing of the nose, correction of the color of the characteristic part, correction of raising the corners of the eyes and the mouth, removal of the fringe line, and the like.

  In addition to the processing described above, correction processing (brightness correction) that makes the skin look fresh, change of skin color, change of hair color, dynamic range expansion, dark part correction, edge unsharpness mask processing, noise removal Such a process may be included in the skin beautification process.

  Thus, the skin beautification process includes a skin area correction process for making wrinkles and rough skin inconspicuous, and a process corresponding to point makeup. The skin beautification process is not only for improving the appearance of a woman, but also because the face is bright and a favorable image can be acquired, it is useful even when the subject is a man or a child.

  Further, the image processing unit 15 performs resizing processing on the main image on which the above-described image processing has been performed, for example, images used when displaying a list of a plurality of images on the display unit 16, and images displayed on the display unit 16. A thumbnail image such as an image (screen nail image) to be used for the generation is generated. Then, the image processing unit 15 executes processing for compressing these images in a JPEG (Joint Photographic Experts Group) format or the like, and processing for decompressing and restoring the compressed images.

  The display unit 16 displays various images. Various images displayed on the display unit 16 are a through image, a main image, a menu image, an image recorded on a recording medium 20 described later, and the like. The operation unit 17 includes a release button 23, a cross key 24, and the like. The release button 23 is operated by the user at the time of shooting. The cross key 24 is operated on the above menu image or the like. The state of the release button 23 and the cross key 24 is detected by the control unit 21, and a sequence based on the detected state of the button or key is executed.

  The memory 18 records a plurality of sample images that are candidates for an image serving as a reference (hereinafter referred to as a reference image) when executing skin beautification processing in the first skin beautifying mode. These sample images are classified for each subject and recorded in the memory 18. In this memory 18, a dedicated folder associated with the subject included in the sample image is recorded. The dedicated folder includes a case where a sample image is newly recorded in the memory 18, a face recognition data obtained from a sample image recorded in the past, and a face recognition data obtained from a newly recorded sample image. Generated if they do not match. In the dedicated folder generated in the memory 18, the above-described sample image and face recognition data acquired from the face area of the subject included in the sample image are recorded. Note that the sample image includes any one of a main image acquired in any one of the shooting mode, the first skin beautification shooting mode, and the second skin beautification shooting mode, and a thumbnail image obtained by resizing the main image. Hereinafter, a case where the main image is recorded as a sample image in a dedicated folder will be described. When the main image acquired in the first skin beautification mode and the second skin beautification mode is used as sample images, correction information indicating the content of skin beautification processing is recorded as one of the supplementary information in these sample images. The face recognition data may be recorded in the supplementary information of the main image, or may be recorded as another file associated with the main image.

  The recording I / F unit 19 writes / reads data to / from the recording medium 20 when the recording medium 20 is connected.

  The control unit 21 performs overall control of the imaging apparatus 10 according to a predetermined sequence program, and executes various calculations (AF, AE, etc.) necessary for shooting. The control unit 21 has functions such as a face detection unit 25, a generation unit 26, and a determination unit 27. The face detection unit 25 extracts feature points from the main image, the sample image, and the image recorded on the recording medium 20 to detect a face region, a face size, and the like. As a method for detecting the face area, a feature point extraction process described in JP-A-2001-16573 is cited. Examples of the feature points include eyebrow, eye, nose, and lip end points, face contour points, head apexes, and chin lower end points.

  The generation unit 26 generates face recognition data indicating the feature amount of the face area from the position of the feature point detected by the face detection unit 25, the relative distance of each feature point, and the like. Note that when a plurality of sample images are recorded in the dedicated folder, the generation unit 26 generates an average of the face recognition data of each sample image as face recognition data.

  The determination unit 27 determines whether a dedicated folder for the subject at the time of shooting has been generated in the memory 18. For example, the determination unit 27 compares the face recognition data of the face area detected from the main image with the face recognition data of the face area detected from the sample image recorded in each dedicated folder, and the subject at the time of shooting The similarity between the face and the face included in the sample image is obtained. Next, the determination unit 27 determines whether or not the obtained similarity exceeds a previously obtained threshold value. If the determination unit 27 determines that the obtained similarity exceeds the threshold, a dedicated folder associated with the subject at the time of shooting is generated in the memory 18 (there is a dedicated folder associated with the subject). Is determined. That is, in this case, the subject at the time of shooting is the same person as the subject included in the sample image recorded in the dedicated folder.

  On the other hand, when the above-described similarity is less than the threshold, the determination unit 27 does not generate the dedicated folder associated with the subject at the time of shooting in the memory 18 (there is no dedicated folder associated with the subject). Is determined. That is, in this case, the subject at the time of shooting is not the same person as the subject included in the sample image recorded in the dedicated folder.

  The bus 22 connects the buffer memory 14, the image processing unit 15, the display unit 16, the recording I / F unit 19, and the control unit 21 to execute input / output of data and signals.

  Next, the operation at the time of photographing will be described based on the flowchart of FIG. 2 is executed when the release button 23 is fully pressed during shooting. Further, hereinafter, for convenience, a scene in which one woman is photographed will be described as an example.

  Step S101 is processing for acquiring the main image. Note that the processing in step S101 is well known, and thus description thereof is omitted here. By executing the processing of step S101, a main image based on the previously set exposure condition is acquired, and the acquired main image is temporarily recorded in the buffer memory.

  Step S102 is processing for generating face recognition data. The face detection unit 25 detects a face area from the main image acquired in step S101. The generation unit 26 generates face recognition data for the face area.

  Step S103 is processing for determining whether or not a dedicated folder for the subject at the time of shooting has been generated in the memory 18. First, the detection unit 25 reads the face recognition data recorded in each dedicated folder in the memory 18. Then, the determination unit 27 compares the face recognition data of the main image with the face recognition data read from each dedicated folder, and obtains the similarity. The determination unit 27 determines that a dedicated folder associated with the subject at the time of shooting is generated in the memory 18 if there is a similarity that exceeds a threshold value among these similarities. In this case, the control unit 21 sets this determination process to YES, and proceeds to step S104.

  On the other hand, the determination unit 27 determines that the dedicated folder associated with the subject at the time of photographing is not generated in the memory 18 when there is no similarity exceeding the threshold among the above-described similarities. In this case, the control unit 21 sets this determination process to NO, and proceeds to step S110.

  As described above, when it is determined YES in the determination process in step S103, that is, when it is determined that the dedicated folder associated with the subject at the time of shooting is generated in the memory 18, the control unit 21 performs step The process proceeds to S104.

  Step S104 is processing to determine whether or not the set shooting mode is the first skin beautification shooting mode. If the set shooting mode is the first skin softening mode, the control unit 21 determines YES in step S104 and proceeds to step S105. On the other hand, if the set photographing mode is not the first skin beautifying mode, the control unit 21 determines NO in step S104 and proceeds to step S110.

  Step S105 is processing to accept selection of a thumbnail image. The control unit 21 reads a thumbnail image from a dedicated folder for the subject at the time of shooting. At the same time, the control unit 21 reads correction information indicating the content of the skin beautifying process.

  As shown in FIG. 3, the display unit 16 displays a list of read thumbnail images in the thumbnail image display area 31. The display unit 16 also displays a frame 32 for displaying the current reference image candidate, a message 33 “Please select a reference image”, and an item “OK” 34. Further, when the correction information is read, the display unit 16 superimposes the correction information on the thumbnail image. For example, when the correction information indicating that the lip area of the thumbnail image 35 is pink is read, the display unit 16 displays the message 36 “Pink lipstick” superimposed on the thumbnail image 35. The control unit 21 receives a selection from the list of thumbnail images via the cross key 24 from the user, and sets a sample image corresponding to the received thumbnail image as a reference image.

  Step S106 is a process for performing a skin beautification process based on the reference image. First, the image processing unit 15 performs exposure correction, soft focus processing, and the like on the face area detected in step S102. Next, the image processing unit 15 performs a skin beautification process on the face area detected from the main image so as to achieve a finish equivalent to the makeup applied to the reference image set in step S106. For example, the control unit 21 identifies skin color, eyebrow shape and color, eye shadow color, eyelash length and color, lipstick color, and the like from the reference image. In addition, when the correction information is read, the control unit 21 identifies these from the correction information. Then, based on the identified information, the image processing unit 15 corrects the skin color, eyebrow shape and color, eye shadow color, eyelash length and color, lipstick color, and the like in the main image. .

  Step S107 is processing to display the main image that has undergone skin beautification processing. The display unit 16 displays the main image that has undergone skin beautification in step S106.

  Step S108 is a process of accepting selection as to whether or not the skin beautification process is appropriate. When the control unit 21 receives a selection from the user via the cross key 24 as to whether or not the skin beautification process is appropriate and receives a selection as appropriate (when the determination in step S108 is YES). Advances to step S109. On the other hand, when the control unit 21 accepts a selection that is not appropriate (when the determination in step S108 is NO), the control unit 21 returns to step S104 and determines whether or not the set photographing mode is the first skin beautifying mode. Determine whether.

  Step S109 is a process of recording the main image on which the skin beautification process has been performed in the dedicated folder and the recording medium 20. The image processing unit 15 performs a resizing process on the main image that has been subjected to the skin beautifying process to generate a thumbnail image. The control unit 21 associates the face recognition data generated in step S102, the main image subjected to the skin beautification process, the thumbnail image, and the correction information, and stores the reference image used for the skin beautification process. Record as a sample image in a folder. At the same time, the data is recorded on the recording medium 20 and the series of processes is terminated.

  If it is determined in step S103 described above that a dedicated folder for the subject at the time of shooting has not been generated, or if it is determined that the shooting mode set in step S104 is not the first skin softening mode, the process proceeds to step S110. move on. Step S110 is processing for determining whether or not the set photographing mode is the second skin beautifying mode. If the set photographing mode is the second skin beautifying mode, the control unit 21 determines YES in step S110 and proceeds to step S111. On the other hand, if the set shooting mode is not the second skin beautification shooting mode, the control unit 21 sets the determination result in step S110 to NO. In this case, the image processing unit 15 performs image processing based on the shooting conditions on the main image. After this image processing, the process proceeds to step S112.

  Step S <b> 111 is a process of performing skin beautification processing based on user settings. First, the image processing unit 15 performs exposure correction, soft focus processing, and the like on the face area detected in step S102. Next, the control unit 21 displays palettes such as skin color, eyebrow shape and color, eye shadow color, eyelash length and color, and lipstick color on the display unit 16 as makeup candidates. Any selection is accepted via the cross key 24. Then, the image processing unit 15 corrects the skin color of the face area, the shape and color of the eyebrows, the color of the eye shadow, the length and color of the eyelashes, the color of the lipstick, and the like based on the received selection.

  Step S112 is a process of displaying the main image that has been subjected to image processing based on the shooting conditions or the main image that has been subjected to skin beautification processing. The display unit 16 displays the main image that has been subjected to image processing based on the shooting conditions or the main image that has been subjected to skin beautification in step S111.

  Step S113 is a process of accepting selection as to whether image processing based on shooting conditions or skin beautification is appropriate. The control unit 21 receives a selection from the user via the cross key 24 as to whether image processing based on the shooting conditions or skin beautification is appropriate, and receives a selection as appropriate (in step S113). If the determination is YES), the process proceeds to step S114. On the other hand, when the selection that it is not appropriate is accepted (when the determination in step S113 is NO), the process returns to step S110 to determine whether or not the set shooting mode is the second shooting mode.

  Step S114 is processing for generating a dedicated folder. The control unit 21 newly creates a dedicated folder in the memory 18.

  Step S115 is a process of recording the main image on which the image processing based on the photographing conditions is performed or the main image on which the skin beautification processing is performed on the dedicated folder and the recording medium 20. When the skin beautification process is not performed in step S111, the image processing unit 15 performs a resizing process on the main image that has been subjected to the image process based on the shooting conditions, and generates a thumbnail image. The control unit 21 associates the main image that has undergone image processing based on the shooting conditions, the thumbnail image, and the face recognition data generated in step S102, and records them as a sample image in the newly generated dedicated folder. At the same time, the data is recorded on the recording medium 20 and the series of processes is terminated. On the other hand, when the skin beautification process is performed in step S111, the image processing unit 15 performs a resizing process on the main image that has been subjected to the skin beautifying process to generate a thumbnail image. The control unit 21 associates the face recognition data generated in step S102, the main image subjected to skin beautification processing, the thumbnail image, and the correction information, and records them as a sample image in a new dedicated folder. At the same time, the data is recorded on the recording medium 20 and the series of processes is terminated.

  In general, makeup that suits a person differs from one person to another. Therefore, an unnatural image may be obtained unless skin-beautifying treatment that suits the person's skin color and facial appearance is performed. In addition, there may be a case where an image acquired when the subject likes the makeup is recorded and the makeup is reproduced with another image. Therefore, the imaging apparatus 10 according to the present embodiment classifies sample images for each subject and records them in a dedicated folder. Then, a reference image is selected from the sample images recorded in the folder dedicated to the subject at the time of shooting, and skin beautification processing is performed on the main image based on the reference image. As a result, the main image can be brought close to the reference image with the desired makeup. Therefore, according to the imaging apparatus 10 of the present embodiment, it is possible to easily generate an image that makes a subject look beautiful and does not give an unnatural impression.

  In the above embodiment, an image including the entire face area of the subject is used as a sample image. However, an image including a part of the face area such as an eyebrow area, an eye area, or a mouth area extracted from the sample image is sampled. It may be an image.

  In the above-described embodiment, an example in which the skin beautification process is performed on all areas of the skin, the eyebrows, the eyes, and the mouth has been described, but the present invention is not limited thereto. For example, the control unit 21 may accept selection of which region to perform skin beautification processing from the user via the cross key 24. Further, the control unit 21 may display the hair color variations as palette candidates on the display unit 16 and accept any selection from the user via the cross key 24.

  In the above embodiment, an example in which selection of a reference image is received from the user via the cross key 24 is shown, but the present invention is not limited to this. For example, the control unit 21 may automatically select the sample image having the highest similarity to the face of the subject at the time of shooting as the reference image, or may automatically select the latest sample image as the reference image. Good. Alternatively, the reference image selected in step S105 by the user may be fixed, and skin softening processing may be performed using the fixed reference image when the process of the flowchart of FIG. 2 is repeated.

  In the above embodiment, when displaying the main image subjected to the skin beautification process on the display unit 16, the correction information may be displayed so that the user can recognize the content of the skin beautification process.

  In the above-described embodiment, a scene in which one woman is photographed is described as an example. However, the present invention is not limited to this. For example, it is possible to correspond to a scene for photographing a man or a child or a scene for photographing a plurality of persons.

  In the above embodiment, whether the subject at the time of shooting is the same person as the subject included in the sample image recorded in the dedicated folder by comparing the face recognition data indicating the feature amount of the face area. Although the example which determines this is shown, it is not restricted to this. For example, it may be determined whether or not they are the same person by well-known pattern matching.

  In the above embodiment, if it is determined in step S103 that the dedicated folder for the subject at the time of shooting has not been generated in the memory 18, it is determined whether or not the set shooting mode is the second skin softening mode. Although the example which determines this is shown, it is not restricted to this. For example, a selection of a sample image recorded in another person's dedicated folder may be accepted, and skin care processing may be performed using the sample image as a reference image. If the subject at the time of shooting and the subject included in the reference image are not the same person, the effect of the skin beautification can be expected, although the quality of makeup is not as good as when the subject is the same person.

  In the above-described embodiment, the target for the skin beautification process is the face area, but the present invention is not limited to this. For example, after changing the aspect ratio of the image to be slightly vertically long, a person portion may be cut out and the size of the image may be adjusted to make the person included in the image look slim.

  In the above-described embodiment, an example in which the skin beautification process is performed on the main image acquired at the time of shooting has been described, but the present invention is not limited thereto. For example, it is possible to perform skin beautification processing on an image recorded on the recording medium 20.

  In the above embodiment, the imaging apparatus 10 has been described as an example, but the present invention is not limited to this. For example, the present invention can be similarly applied to an image processing device such as a personal computer, a portable terminal device such as a mobile phone having an imaging function, and an image display device such as a digital photo frame. Further, the functions shown in the image processing unit 15 and the control unit 21 in FIG. 1 and the flow of the flowchart shown in FIG. 2 are recorded as a program on a computer-readable recording medium, and the program is executed by the computer. Good.

DESCRIPTION OF SYMBOLS 10 ... Imaging device, 12 ... Image sensor, 15 ... Image processing part, 16 ... Display part, 17 ... Operation part, 21 ... Control part, 25 ... Face detection part, 26 ... Generation part, 27 ... Determination part

Claims (11)

An image input unit for inputting the first image;
A face detection unit for detecting a face region of the first image input by the image input unit;
Based on a second image that is different from the first image and has a feature quantity that is the same as the feature quantity of the face area of the first image detected by the face detector, the face image of the first image An image processing apparatus comprising: an image correction unit that executes correction processing. The image processing apparatus according to claim 1.
A display unit that displays one or a plurality of the second images when the first image is corrected by the image correction unit;
An image processing apparatus comprising: a selection unit that selects a reference image serving as a reference for the correction process from the second image displayed on the display unit. The image processing apparatus according to claim 2,
An image storage unit for storing one or a plurality of the second images;
An image processing apparatus comprising: a selection unit that selects a reference image serving as a reference for the correction process from the second image stored in the storage unit. The image processing apparatus according to claim 3.
The image processing apparatus, wherein the second image is stored in the image storage unit together for each subject included in the second image. The image processing apparatus according to claim 1, wherein:
The image processing apparatus, wherein the second image is an image having at least a part of a face area. The image processing apparatus according to any one of claims 1 to 5,
The first image corrected by the image correction unit is stored in association with correction information indicating the content of the correction process. The image processing apparatus according to claim 6.
An image processing apparatus, wherein the first image associated with the correction information is stored in the image storage unit as the second image. The image processing apparatus according to any one of claims 1 to 7,
The image processing apparatus according to claim 1, wherein the correction processing includes at least one of color tone correction, contrast correction, brightness correction, and correction of a facial feature part.   An imaging apparatus comprising the image processing apparatus according to any one of claims 1 to 8. An image input step of inputting the first image;
A face detection step of detecting a face region of the first image input by the image input step;
Based on a second image that is different from the first image and has a feature quantity that is the same as the feature quantity of the face area of the first image detected by the face detector, the face image of the first image An image processing program capable of executing an image correction process for executing correction processing by a computer.   A computer-readable storage medium storing the image processing program according to claim 10.

Images