JP2010092094A - Image processing apparatus, image processing program, and digital camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing program and a digital camera providing images favorable as portrait photos. <P>SOLUTION: The image processing apparatus includes a subject extraction unit for extracting a main subject from an image, and determining the range of a background area in the image not including the main subject, an evaluation value calculating unit for evaluating the expression of the main subject, and calculating an expression evaluation value indicating the degree of the expression, and an image processing unit which performs a blurring process on the background area according to the expression evaluation value. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing, an image processing program, and a digital camera including them.

2. Description of the Related Art Conventionally, a digital camera that recognizes a facial expression of a subject and performs image processing based on the recognition result is known. For example, the digital camera disclosed in Patent Document 1 performs image brightness correction, text or character copying, and the like according to the facial expression of the subject.
JP 2001-51338 A

  By the way, in portrait photography, a photographic expression that blurs the background and highlights the main subject is preferred. However, since the above-described digital camera does not blur the background, a preferable image cannot always be obtained.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing program, and a digital camera that acquire a preferable image as a portrait photograph.

  The image processing apparatus of the present invention extracts a main subject from an image, evaluates a facial expression of the main subject, a subject extraction unit that determines a range of a background area that does not include the main subject in the image, An evaluation value calculation unit that calculates a facial expression evaluation value indicating the degree of the image, and an image processing unit that performs a blurring process on the background area according to the facial expression evaluation value.

  Preferably, the evaluation value calculation unit may calculate an evaluation value indicating the degree of smile as the facial expression evaluation value, and the image processing unit may increase the blurring degree as the facial expression evaluation value is higher.

  Preferably, the image processing unit may increase the luminance of the background region as the facial expression evaluation value is higher.

  Preferably, the image processing unit may further include an analysis unit that analyzes a spatial frequency of the background region, and the degree of blurring may be increased as the number of high-frequency components increases.

  Preferably, the image processing unit further includes a ratio calculation unit that calculates a ratio of the face area of the main subject to the image, and the image processing unit increases the degree of blurring when the ratio of the face area is equal to or greater than a threshold value. May be.

  Preferably, the image processing unit may further include a luminance calculation unit that calculates the luminance of the background region, and the degree of blurring may be increased as the luminance is lower.

  Preferably, the image processing unit further includes an estimation unit that estimates at least one of the age and sex of the main subject, and the image processing unit uses the estimation unit to determine whether the age of the main subject is equal to or lower than a threshold value, or the main subject is a female. When it is estimated that, the degree of blurring may be increased.

  Preferably, the image processing unit is reddish with respect to a face area of the main subject when the estimation unit estimates that the age of the main subject is equal to or less than a threshold value or the main subject is a woman. Color correction to increase the color may be performed.

  Preferably, the evaluation value calculation unit may calculate an evaluation value indicating the degree of smile as the facial expression evaluation value, and the image processing unit may change a blur shape according to the facial expression evaluation value.

  Preferably, the image processing unit may further include an estimation unit that estimates at least one of age and sex of the main subject, and the image processing unit may change a blur shape according to an estimation result of the estimation unit.

  Preferably, the evaluation value calculation unit calculates an evaluation value indicating a degree of anger as the facial expression evaluation value, and the image processing unit determines that the main subject is in the case where the facial expression evaluation value is greater than or equal to a threshold value. Only the upper part of the head may be blurred.

  Preferably, the evaluation value calculation unit calculates an evaluation value indicating a degree of sadness as the facial expression evaluation value, and the image processing unit determines that the background region has a background value when the facial expression evaluation value is equal to or greater than a threshold value. You may perform the color correction which makes a color tone a monotone or a sepia tone.

  Preferably, the image processing unit further includes a display unit that displays an image of a predetermined facial expression among the images, and a selection unit that selects any one of the images displayed on the display unit. Blur processing may be performed on the image selected by the selection unit.

  A digital camera according to the present invention is a digital camera including an imaging unit that acquires an image by imaging a main subject and the image processing apparatus according to the first invention, wherein the image processing unit includes at least the imaging Blur processing is performed on the image acquired by the section.

  Preferably, an imaging control unit that controls the imaging unit according to the facial expression of the main subject may be further provided.

  An image processing program of the present invention is an image processing program for realizing image processing on image data to be processed by a computer, and extracts a main subject from an image and does not include the main subject in the image. A subject extraction step for determining a range of a background region; an evaluation value calculation step for evaluating a facial expression of the main subject and calculating a facial expression evaluation value indicating a degree of facial expression; and the facial expression evaluation value for the background region. And an image processing step for performing a blurring process accordingly.

  According to the image processing apparatus, the image processing program, and the digital camera of the present invention, a preferable image can be acquired as a portrait photograph.

(First Embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In this embodiment, a digital camera is used as an example of the image processing apparatus of the present invention. FIG. 1 is a block diagram showing the configuration of the digital camera 1 according to the first embodiment of the present invention. As shown in FIG. 1, the digital camera 1 includes an imaging lens 2, an imaging device 3, an A / D conversion unit 4, a buffer memory 5, an image processing unit 6, a compression / decompression processing unit 7, a control unit 10, a display unit 14, A memory 15, an operation unit 16, a recording I / F unit 19, a recording medium 20, and a bus 21 are provided.

  The imaging lens 2 forms a subject image on the imaging surface of the imaging element 3. The image sensor 3 photoelectrically converts a subject image formed by a light beam that has passed through the imaging lens 2 and outputs analog image signals corresponding to the R, G, and B colors. In the shooting mode, the image sensor 3 performs thinning readout at predetermined intervals to capture a through image. Thereafter, the image sensor 3 captures the main image when the release button 17 is fully pressed. Further, the imaging device 3 automatically captures the main image when the facial expression recognition unit 12 recognizes a facial expression of smile, anger, and sadness from the through image in each facial expression mode of smile, anger, and sadness, which will be described later. You can also

  An image signal output from the image sensor 3 is input to the A / D converter 4. The A / D converter 4 performs A / D conversion on the analog image signal output from the image sensor 3 and converts the analog image signal into a digital image signal. The digital image signals are collected into one frame and recorded in the buffer memory 5 as image data. The buffer memory 5 temporarily records image data in the pre-process and post-process of image processing by the image processing unit 6.

  The image processing unit 6 performs image processing on the image data stored in the buffer memory 5. Examples of the image processing include well-known white balance adjustment, color interpolation, gradation conversion processing, and contour enhancement processing. Further, the image processing unit 6 performs a blurring process to be described later on the main image and the image recorded on the recording medium 20. The compression / decompression processing unit 7 includes an analysis unit 8 and a compression unit 9. The analysis unit 8 performs spatial frequency analysis on the image data. The compression unit 9 executes compression / decompression processing in JPEG (Joint Photographic Experts Group) format.

  The control unit 10 performs overall control of the digital camera 1 according to a predetermined sequence program, and executes various calculations (AF, AE, etc.) necessary for imaging. The control unit 10 includes a face detection unit 11, a facial expression recognition unit 12, and a subject extraction unit 13. The face detection unit 11 extracts feature points from the through image, the main image, and the image recorded on the recording medium 20, and detects the face area, the face size, and the like of the main subject A described later. The facial expression recognition unit 12 recognizes the facial expression detected by the face detection unit 11. The subject extraction unit 13 extracts the main subject A from the main image and the image recorded on the recording medium 20, and determines the range of the background region B that does not include the main subject A, which will be described later.

  Further, the control unit 10 calculates a facial expression evaluation value indicating the degree of facial expression. For example, the control unit 10 calculates, as facial expression evaluation values, a smile evaluation value indicating the degree of smile, an anger evaluation value indicating the degree of anger, and a sadness evaluation value indicating the degree of sadness. The memory 15 stores in advance a smile evaluation reference table, an anger evaluation reference table, and a sadness evaluation reference table that are used by the control unit 10 to calculate a smile evaluation value, an anger evaluation value, and a sadness evaluation value, respectively. Each facial expression evaluation value and each evaluation reference table will be described later.

  The display unit 14 displays various images under the control of the control unit 10. Various images displayed on the display unit 14 include a through image, a main image, an image recorded on the recording medium 20, a menu image, and the like. The operation unit 16 includes a release button 17 and an operation button 18. The release button 17 is operated by the user during imaging. The operation button 18 is operated with the menu image or the like. The state of the release button 17 and the operation button 18 is detected by the control unit 10, and a sequence based on the detected button state is executed. The recording I / F unit 19 includes a connector for connecting the recording medium 20. When the recording I / F unit 19 and the recording medium 20 are connected, data writing / reading is executed on the recording medium 20. The bus 21 connects the buffer memory 5, the image processing unit 6, the compression / decompression processing unit 7, the control unit 10, the display unit 14, the memory 15, and the recording I / F unit 19, thereby inputting and outputting data and signals. Execute. The output of the A / D converter 4 is connected to the image processor 6 via the bus 21.

  Here, the blurring process will be described. The blurring process of the present embodiment is a process of expanding a bright spot using a point spread function. FIG. 2 is a diagram illustrating the luminance distribution of the bright spots before and after the blurring process. The vertical axis in FIG. 2 indicates the luminance, and the horizontal axis indicates the spread from the luminance center.

  As shown in FIG. 2, the image processing unit 6 performs the blurring process so that the luminance information of the bright spots is the same before and after the blurring process. In FIG. 2, the luminance information is the area of the bright spot before the blurring process, that is, the area of the area E that is shaded, and the area of the bright spot after the blurring process, that is, the area F of the area F that is hatched. It corresponds to the area. The image processing unit 6 can obtain an image expressed so that the main subject A emerges by performing a blurring process on the background region B. In the present embodiment, the width dimension of the portion where the luminance is ½ of the peak is used as an index of the blurring degree. For example, the blurring degree in FIG.

  Further, the image processing unit 6 can change the way in which the bright spots are spread (hereinafter referred to as a blur shape). The blur shape has variations in the shape of a bright spot spread on a two-dimensional image and the attenuation of luminance from the center to the periphery. FIG. 3 is a diagram illustrating an example of a blurred shape. As shown in FIG. 3, the types of blur shapes include “Circle 1”, “Circle 2”, “Star”, etc., and the blur shape indicating “Circle 1” has a bright spot spread in a circular shape. In addition, the shape is blurred so that the outline of the bright spot becomes clear. Hereinafter, the blurred shape of “circle 1” will be described with reference numeral 31. The blur shape indicating “Circle 2” has a shape in which the bright spot portion spreads in a circular shape and the brightness decreases in a gradation shape from the center of the bright spot toward the radial direction. Hereinafter, the blurring shape of “circle 2” will be described with reference numeral 32. The blur shape indicating “star” is a shape in which the bright spot portion spreads in a star shape and is blurred so that the outline of the bright spot becomes clear. Hereinafter, the “star” blur shape will be described with reference numeral 33.

  An image in which the background region B of the main subject A is blurred differs depending on the situation of the main subject A and the background region B. For example, in the case of a bright shooting scene such as outdoors in the daytime on a sunny day, the main subject A appears more clearly by using a blurring shape with an unclear outline such as the blurring shape 32 shown in “Circle 2”. It is possible to create a blurred image that is emphasized and has a little uncomfortable appearance. On the other hand, in the case of a dark scene such as a night view, the bright spot of the background region B is emphasized and more impressive when a blur shape with a clear outline such as the blur shape 31 shown in “Circle 1” is used. A blurred image can be created. In the case where the main subject A is an image of a child, if the blurring shape 33 shown by “star” is used, a fun and fairy tale effect can be produced, and the blurring process works effectively.

  The image processing unit 6 changes the blurring degree and the blurring shape by using different point spread functions. The same blurring process can be performed by using various digital filters such as a low-pass filter instead of the point spread function.

  FIG. 4 is a flowchart showing the operation of the control unit 10 during imaging in the smile mode in the digital camera 1. Note that, in each of the following facial expression modes, for the sake of convenience, description will be made on the premise of a scene where one person is imaged.

  In step S101, the control unit 10 starts the smile mode. For example, the control unit 10 controls the imaging element 3 to start capturing a through image.

  In step S102, the face detection unit 11 detects a face area from the through image. For example, the face detection unit 11 extracts a face region by a feature point extraction process described in Japanese Patent Application Laid-Open No. 2001-16573. Examples of the feature points include eyebrow, eye, nose, and lip end points, face contour points, head apexes, and chin lower end points. Next, the facial expression recognition unit 12 recognizes the facial expression detected by the face detection unit 11. For example, as shown in JP-A-8-249453 and JP-A-10-255043, the facial expression recognition unit 12 has a total of six basic types of facial expressions such as anger, disgust, fear, sadness, smile, and surprise. Consider facial expressions, process time-series images from non-facial expressions to basic facial expressions, and recognize facial expressions.

  In step S103, the control unit 10 determines whether or not the facial expression recognized in step S102 is a smile. When it is a smile (when determination of step S103 becomes YES), it progresses to step S104. On the other hand, when it is not a smile (when determination of step S103 becomes NO), it returns to step S102.

  In step S <b> 104, the control unit 10 controls the image sensor 3 to capture the main image. Then, the control unit 10 controls the image processing unit 6 to perform image processing on the main image data and store it in the buffer memory 5. FIG. 5A shows an example of a main image in which a child as the main subject A and trees as the background area B are captured.

  In step S105, the subject extraction unit 13 extracts the main subject A from the main image captured in step S104 according to the following procedure, and determines the range of the background region B that does not include the main subject A.

  First, the subject extraction unit 13 divides the main image into a plurality of regions by contour lines in the main image. For example, the subject extraction unit 13 divides the main image by edge extraction processing by the image processing unit 6. In this case, the image processing unit 6 performs edge extraction processing using a differential filter (such as a Laplacian filter) on the main image data. Then, the subject extraction unit 13 divides the main image into a plurality of regions based on the extracted contour line.

  Alternatively, the subject extraction unit 13 can also divide the main image into a plurality of regions based on the output of the analysis unit 8. In this case, the subject extraction unit 13 divides the main image into pixel blocks of about 8 × 8 pixels. Next, the analysis unit 8 performs a discrete cosine transform (DCT) process on each of the pixel blocks, and acquires a DCT coefficient for each pixel block.

  Here, the relationship between the DCT coefficient and the extraction process of the main subject A will be briefly described. An image with a clear outline contains many high frequency components in the image data. Therefore, the DCT coefficient obtained from an image with a clear outline has a non-zero value. On the other hand, an image that does not include a contour has fewer high-frequency components than an image with a clear contour. Therefore, the DCT coefficient obtained from the image not including the contour becomes 0 from the low frequency component.

  Therefore, the subject extraction unit 13 pays attention to the distribution of the DCT coefficient for each frequency component, and extracts the pixel block whose DCT coefficient corresponding to the high frequency component is not zero as the contour of the subject. Thereafter, the subject extraction unit 13 divides the main image into a plurality of regions based on the extracted pixel blocks. Note that the subject extraction unit 13 may group the areas partitioned by the outline based on information such as color difference and brightness of the main image.

  Secondly, the face detection unit detects a face region from the main image. Then, the subject extraction unit 13 determines the range of the main subject A and the background region B from the region of the main image divided above based on the position of the face region. FIG. 5B is a diagram illustrating a state in which the main image in FIG. 5A is divided into the main subject A and the background region B. In FIG. 5B, the hatched area is the background area B.

  In step S106, the control unit 10 calculates an evaluation point of the shape of each element of the eyebrows, pupils, and lips of the face area with reference to the smile evaluation table as disclosed in JP 2004-46591A, The smile evaluation value is calculated by adding the weights obtained by weighting each evaluation point. The smile evaluation criterion table corresponds to an element area that stores data representing face components (for example, eyebrows, eyes, and lips) and a coefficient area that stores data representing coefficients of corresponding face components. A reference image of the constituent elements of the face to be operated and an evaluation point area for storing the evaluation points. The smile evaluation value is 100 points, and the closer to a full smile, the higher the score.

  In step S107, the control unit 10 estimates the age and gender of the main subject A using the inference process described in Japanese Patent Application Laid-Open No. 2007-280291. And the control part 10 determines with it being a child, when the age of the main subject A is below a threshold value.

  In step S108, the control unit 10 calculates the ratio of the face area of the main subject A to the main image. And the control part 10 determines with it being a bust shot, when the ratio of a face area is more than a threshold value (for example, 10%).

  In step S109, the control unit 10 calculates the luminance of the background region B. For example, the control unit 10 calculates the luminance for each pixel for all the pixels constituting the background region B. Then, the control unit 10 divides the sum of the luminances by the number of pixels in the background region B, and calculates an average value of the luminances of all the pixels constituting the background region B.

  In step S110, the control unit 10 obtains the degree of complexity of the background region B. When the background region B is complicated, the image data includes many high frequency components. Therefore, the control unit 10 controls the analysis unit 8 to perform the same processing as step S105, and the DCT coefficient corresponding to the high frequency component is obtained from the background region B divided into pixel blocks of about 8 × 8 pixels. Pixel blocks that are not zero values are extracted. And the control part 10 calculates | requires the grade of the complexity of the background area | region B according to the number of the extracted pixel blocks.

  In step S111, the control unit 10 determines the blurring degree and the blurring shape by combining the following (a) to (g). Thereby, since the blurring process suitable for the smile of the main subject A can be performed, an image preferable as a portrait photograph can be obtained.

  (A) The control unit 10 increases the blurring degree as the smile evaluation value calculated in step S106 increases. The relationship between the smile evaluation value and the blurring degree is shown in FIG. As shown in FIG. 6 (1), as the degree of blur increases, the area having the luminance substantially equal to the luminance at the center of the bright spot (hereinafter referred to as an equal luminance area) becomes narrower. The brightness change becomes gentle. On the other hand, as the degree of blurring becomes weaker, the equiluminance region becomes wider, and the luminance change in the peripheral portion of the equiluminance region becomes steeper. As a result, an image with a larger smile evaluation value and closer to a full smile is selected to have a strong blurring degree such that the brightness at the center of the bright spot is lower and the attenuation from the center to the peripheral part is gentler. Is an emphasized image. On the other hand, when the smile evaluation value is small, the weak blurring degree is selected, and the enhancement of the main subject A by the blurring process is suppressed.

  (B) The control unit 10 increases the degree of blurring as the luminance of the background region B calculated in step S109 is lower. FIG. 6B shows the relationship between the brightness of the background region B and the degree of blurring. Since an image with low brightness in the background region B does not stand out due to the blurring process, the strong blurring process can provide an image with a large effect of enhancing the main subject A and with a little uncomfortable feeling.

  (C) When it is estimated that the main subject A is a child or a woman in step S107, the control unit 10 increases the degree of blurring. This is because when the main subject A is an image of a child or a woman, an image that emphasizes the main subject A by strong blurring processing is preferred.

  (D) When the control unit 10 determines in step S108 that the shot is a bust shot, the control unit 10 increases the degree of blurring. This is because, in an image in which the main subject A is large, such as a bust shot, the higher the degree of blurring, the more effective the operation in favoring the impression of the image. On the other hand, if the main subject A is a small image and the background region B is blurred, the image is mostly blurred and does not necessarily give a favorable impression.

  (E) The control unit 10 increases the degree of blur as the background region B obtained in step S110 has a higher degree of complexity. This is because the more complex the background region B, the higher the possibility that the main subject A will not stand out.

  (F) When the main subject A is a woman, the control unit 10 uses the blur shape 31 indicating “circle 1”, and when the main subject A is a child, the control unit 10 indicates a blur shape 33 indicating “star”. Is used. On the other hand, when the main subject A is an adult male, the blur shape 32 indicating “circle 2” is used. The relationship between the type of the main subject A and the blur shape is shown in FIG.

  When the main subject A is a woman, if the blurring shape 31 indicating “circle 1” is used, the bright spot of the background region B clearly spreads, and an impressive effect is obtained. In the case where the main subject A is a child, if the blur shape 33 indicating “star” is used, a fun and fairy tale effect can be produced and blur processing can be effectively performed. In addition, when the main subject A is a man, the production effects such as “Circle 1” and “Star” are often not suitable for the image of the man, but by using the blur shape 32 indicating “Circle 2”, An image in which the main subject A is emphasized without a sense of incongruity can be obtained.

  (G) When the smile evaluation value is 70 points or more, the control unit 10 uses the blurring shape 31 indicating “circle 1”, and when the smile evaluation value is less than 70 points, “circle 2”. A blur shape 32 indicating the above is used. The relationship between the smile evaluation value and the blurred shape is shown in FIG. When the smile evaluation value is large, the main subject A of the smile is more effectively enhanced by the blurring process in which the bright spot of the background region B is clearly expanded by the blur shape 31 indicating “circle 1”. . On the other hand, when the smile evaluation value is small, it is possible to obtain an image in which the main subject A is emphasized while preventing an impression of an excessive effect by using the blurring shape 32 indicating “circle 2”.

  In step S112, the control unit 10 controls the image processing unit 6 to perform a blurring process on the background region B using the blurring degree and blurring shape determined in step S111 so that the main subject A emerges. Get the image expressed in. An example of the main image after the blurring process is shown in FIG. In the example of FIG. 5 (3), since the main subject A is a child, a blur shape 33 indicating “star” is used. Thereby, since the background area | region B becomes fairy tale, childhood can be emphasized.

  In step S113, the control unit 10 controls the image processing unit 6 to produce an effect suitable for the smile of the main subject A, and increases the brightness of the background region B as the smile evaluation value calculated in step S106 increases. . Further, the control unit 10 controls the image processing unit 6 to perform color correction that increases redness on the face area when the main subject A is a child or a woman. Thereby, childhood or femininity can be emphasized more. When the smile evaluation value is 90 points or more, color processing may be performed so as to erase the subject reflected in the background area B. For example, in the example of FIG. 5 (3), since the background area B is trees, the entire background area B may be interpolated in green and processed into a uniform background. If the background region B is a uniform color, an image in which the main subject A is more prominent is obtained.

  In step S114, the control unit 10 records the main image data subjected to the image processing in step S113 as an image file on the recording medium 20, and ends a series of processing. In addition, it is preferable that the control unit 10 records the smile evaluation value, the blurring degree, and the blurring shape together with the main image data in the image file as information such as a tag.

  Next, the blurring process in the anger mode will be described. FIG. 7 is a flowchart illustrating the operation of the control unit 10 during imaging in the anger mode in the digital camera 1.

  In step S201, the control unit 10 starts the anger mode. For example, the control unit 10 controls the imaging element 3 to start capturing a through image.

  In step S202, the face detection unit 11 detects a face area from the through image. Next, the facial expression recognition unit 12 performs the same processing as in step S102 of FIG. 4 to recognize the facial area expression detected by the face detection unit 11.

  In step S203, the control unit 10 determines whether the facial expression recognized in step S202 is angry. If it is angry (if the determination in step S203 is YES), the process proceeds to step S204. On the other hand, when it is not anger (when determination of step S203 becomes NO), it returns to step S202.

  In step S <b> 204, the control unit 10 controls the image sensor 3 to capture the main image. Then, the control unit 10 controls the image processing unit 6 to perform image processing on the main image data and store it in the buffer memory 5.

  In step S205, the subject extraction unit 13 performs the same processing as in step S105 in FIG. 4, extracts the main subject A from the main image captured in step S204, and determines the range of the background region B.

  In step S206, the control unit 10 performs the same process as in step S106 of FIG. 4 and calculates the anger evaluation value using the anger evaluation reference table stored in advance in the memory 15. Note that the anger evaluation criterion table has the same configuration as the smile evaluation criterion table described in step S106. The anger evaluation value is 100 points, and the higher the anger level, the higher the score.

  In step S207, the control unit 10 controls the image processing unit 6 to perform the blurring process only on the upper part of the head of the main subject A in the background region B. Note that the control unit 10 may increase the blurring degree as the anger evaluation value calculated in step S206 is higher. Thus, steam can be reproduced as if standing from the head, so that the angry expression of the main subject A can be further enhanced.

  In step S208, the control unit 10 controls the image processing unit 6 to perform color correction that increases redness on the face area. Thereby, the emotion of anger can be reproduced realistically as an image.

  In step S209, the control unit 10 records the main image data subjected to color correction in step S208 on the recording medium 20 as an image file, and ends a series of processes. In addition, it is preferable that the control part 10 records an anger evaluation value and a blurring degree with information such as a tag in the image file together with the main image data.

  Next, the blurring process in the sadness mode will be described. FIG. 8 is a flowchart illustrating the operation of the control unit 10 during imaging in the sadness mode in the digital camera 1.

  In step S301, the control unit 10 starts a sadness mode. For example, the control unit 10 controls the imaging element 3 to start capturing a through image.

  In step S302, the face detection unit 11 detects a face area from the through image. Next, the facial expression recognition unit 12 performs the same processing as in step S102 of FIG. 4 to recognize the facial area expression detected by the face detection unit 11.

  In step S303, the control unit 10 determines whether the facial expression recognized in step S302 is sad. If it is sad (if the determination in step S303 is YES), the process proceeds to step S304. On the other hand, when it is not sadness (when determination of step S303 becomes NO), it returns to step S302.

  In step S304, the control unit 10 controls the image sensor 3 to capture the main image. Then, the control unit 10 controls the image processing unit 6 to perform image processing on the main image data and store it in the buffer memory 5.

  In step S305, the subject extraction unit 13 performs processing similar to that in step S105 in FIG. 4, extracts the main subject A from the main image captured in step S304, and determines the range of the background region B.

  In step S306, the control unit 10 performs the same process as in step S106 in FIG. 4 and calculates a sadness evaluation value using a sadness evaluation reference table stored in the memory 15 in advance. Note that the sadness evaluation criterion table has the same configuration as the smile evaluation criterion table described in step S106. The sadness evaluation value is 100 points, and the higher the degree of sadness, the higher the score.

  In step S307, the control unit 10 controls the image processing unit 6 to perform a blurring process on the background region B, and obtain an image expressed so that the main subject A appears. Note that the control unit 10 may increase the blurring degree as the sadness evaluation value calculated in step S306 increases. Thereby, the sad expression of the main subject A can be further enhanced.

  In step S308, the control unit 10 controls the image processing unit 6 to perform color correction for increasing bluish color on the face area. Thereby, the emotion of sadness can be reproduced as an image.

  In step S309, the control unit 10 determines whether the sadness evaluation value calculated in step S306 is greater than or equal to a threshold value. If the sadness evaluation value is greater than or equal to the threshold (if the determination in step S309 is YES), the process proceeds to step S310. On the other hand, when the sadness evaluation value is less than the threshold value (when the determination in step S309 is NO), the process proceeds to step S311.

  In step S310, the control unit 10 controls the image processing unit 6 to perform color correction so that the color tone of the background region B is monotone or sepia. As a result, it is possible to reproduce an image of a feeling of sorrow.

  If the determination in step S309 is YES in step S311, the control unit 10 records the main image data subjected to color correction in step S308 as an image file on the recording medium 20, and the determination in step S309 is NO. In this case, the main image data subjected to the color correction in step S310 is recorded on the recording medium as an image file, and a series of processes is completed. The control unit 10 preferably records the sadness evaluation value and the blurring degree in the image file as information such as a tag together with the main image data.

  The digital camera 1 of the first embodiment enhances the main subject A by performing image processing suitable for the expression of the main subject A. Therefore, according to the digital camera 1 of the first embodiment, it is possible to obtain a preferable image as a portrait photograph.

  Further, since the digital camera 1 of the first embodiment automatically captures the main image by recognizing the facial expression of the main subject A, an image of the main subject A having a desired expression is acquired without missing a photo opportunity. can do.

  In the first embodiment, an example is shown in which the main image is automatically captured when a predetermined facial expression is recognized by the facial expression recognition unit 12 in each facial expression mode. However, the present invention is not limited to this. For example, in each facial expression mode, the main image may be captured by pressing the release button 17 fully regardless of facial expression recognition. Further, in the shooting mode, the processing after step S105 in FIG. 4, the processing after step S205 in FIG. 7, or the processing after step S305 in FIG. 8 is performed on the main image captured by fully pressing the release button 17. You may perform the process of.

  Further, face recognition data may be generated based on the feature points of the face area detected by the face detection unit 11 and stored in the memory 15. In this case, the control unit 10 performs face recognition processing for determining whether the face of the person in the through image is the face of the registered person. Then, the control unit 10 automatically captures the main image when it determines that it matches the registered person and recognizes a predetermined facial expression. Furthermore, it may be configured such that a facial expression recognition level can be set for each registered person. For example, a person who usually does not laugh very much can recognize that even a little laughing is recognized as a smile and automatically capture the main image.

  In step S207, the upper portion of the head of the main subject A is subjected to the blurring process, but the present invention is not limited to this. For example, “ugofy” v1.0.0.1, which processes an image in an animated manner that sways, may be used. Thereby, since the upper part of the head of the main subject A can be moved, the angry expression of the main subject A can be further enhanced.

  Further, the extraction method of the main subject A by the subject extraction unit 13 is not limited to the method described above. For example, as disclosed in Japanese Patent Laid-Open No. 2002-44512, the control unit 10 controls the image sensor 3 to capture a preliminary image when capturing a main image in step S104. Then, the subject extraction unit 13 may compare the luminance of the main image and the preliminary image and separate the main subject A and the background region B from the main image based on the result.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. In the first embodiment, an example in which the blurring process is performed on the main image at the time of imaging is shown. In the second embodiment, an example in which blurring processing is performed on an image recorded on the recording medium 20 during reproduction will be described. Here, since the configuration of the digital camera in the following embodiment is the same as that of the digital camera 1 of the first embodiment shown in FIG.

  FIG. 9 is a flowchart showing the operation of the control unit 10 during reproduction in the digital camera 1 of the second embodiment. For convenience, the description will be made on the assumption that an image of one person is captured.

  In step S401, the control unit 10 starts the reproduction mode. For example, the control unit 10 displays a menu image 41 on which a plurality of items relating to the retouch function and an item “smile background blur” are displayed on the display unit 14 (see FIG. 10A). Here, “smile background blur” is an item indicating a function of blurring a smiley image.

  In step S <b> 402, the control unit 10 receives selection of an item “smile background blur” from the user via the operation button 18. In addition, when the “Smile background blur” item is selected, the control unit 10 “processes the subject to make the subject more attractive by blurring the background. Your smile will be more attractive.” May be displayed on the display unit 14.

  In step S403, the control unit 10 reads a smile evaluation value from information such as a tag recorded in the image file of the recording medium 20, and reads an image having a smile evaluation value equal to or greater than a threshold value as a smile image. Then, the control unit 10 displays a list of thumbnail images corresponding to the read image in the thumbnail display area 42 of the display unit 14 as illustrated in FIG. If the smile evaluation value is not recorded on the recording medium 20, the face detection unit 11 detects a face area from the image recorded on the recording medium 20. Then, the facial expression recognition unit 12 recognizes a facial expression from the face area detected by the face detection unit 11 and reads a smile image.

  In step S404, the control unit 10 accepts selection of any image from the list of thumbnail images displayed in step S403 via the operation button 18 from the user. Thereafter, as shown in FIG. 10 (3), the control unit 10 accepts a selection as to whether or not to perform the blurring process on the accepted image.

  Steps S405 to S413 correspond to steps S105 to S113 in FIG.

  In step S414, the control unit 10 displays the image 43 that has been subjected to the image processing in step S413 on the display unit 14, as illustrated in FIG. The control unit 10 also displays a slider bar 44 and a slider 45 indicating the degree of blur by sliding in the slider bar 44. Note that the more the slider 45 is on the right, the stronger the blurring degree. Further, the user can customize the degree of blur by changing the position of the slider 45 via the operation button 18.

  In step S415, the control unit 10 determines whether or not a change in the position of the slider 45 has been received from the user via the operation button 18. When the change of the position of the slider 45 is accepted (when the determination in step S415 is YES), the process returns to step S412 and the blurring process is performed again so that the blurring degree according to the position of the slider 45 is obtained. When the change of the position of the slider 45 is not accepted (when the determination in step S415 is NO), the process proceeds to step S416.

  In step S416, the control unit 10 receives a selection as to whether or not to record the image 43 displayed in step S414 from the user via the operation button 18. When the image 43 is recorded (when the determination in step S416 is YES), the process proceeds to step S417. On the other hand, when the image 43 is not recorded (when the determination in step S416 is NO), the processing in FIG. 9 ends.

  In step S417, the control unit 10 records the image data subjected to the image processing in step S413 on the recording medium 20, and ends a series of processing.

  In the second embodiment, the example in which the blurring process is performed on the smiley image has been described. However, the second embodiment can be applied to an image of an angry expression and an image of a sadness. For example, the control unit 10 accepts the selection of one of the images of the angry facial expression, and performs the processing after step S205 in FIG. Thereafter, the control unit 10 may accept a change in the degree of blurring. Further, the control unit 10 accepts the selection of any image from the images of the sad expression, and performs the processing after step S305 in FIG. Thereafter, the control unit 10 may accept a change in the degree of blurring.

  The digital camera 1 of the second embodiment enhances the main subject A by performing image processing suitable for the expression of the main subject A on the image recorded on the recording medium 20. Therefore, according to the digital camera 1 of the second embodiment, a preferable image can be acquired as a portrait photograph, as in the first embodiment.

In addition, since the digital camera 1 according to the second embodiment accepts a change in the degree of blurring, it is possible to acquire an image with a degree of blurring according to the user's preference.
(Supplement of embodiment)
(1) In the first embodiment and the second embodiment, the example of the digital camera has been described, but the present invention is not limited to this. For example, a program installed in a computer can be provided with functions shown in a procedure for calculating facial expression evaluation values and a blurring procedure in the first and second embodiments. In this case, it is possible to store the image processing program in a storage medium in advance and execute or install the image processing program after connecting to the computer.

  (2) In the first embodiment and the second embodiment, the blurring process is performed by expanding the bright spots, but the present invention is not limited to this. For example, low pass filter processing may be used.

  (3) In the first embodiment and the second embodiment, three types of blur shapes are shown: a blur shape 31 indicating “circle 1”, a blur shape 32 indicating “circle 2”, and a blur shape 33 indicating “star”. However, it is not limited to this. For example, a blur shape indicating “heart” may be used. The blurring shape indicating the “heart” has a shape in which the bright spot portion spreads in a heart shape and is blurred so that the outline of the bright spot becomes clear.

  (4) In the first embodiment and the second embodiment, the example in which the above (a) to (g) are combined as elements for determining the blurring degree and the blurring shape has been described, but the blurring degree and the blurring shape are determined. The element is not limited to this.

  (5) A procedure in which the face detection unit 11 specifies the main subject A when the digital camera 1 of the first embodiment captures a scene including a plurality of persons will be described. For example, the face detection unit 11 automatically specifies the main subject A from a plurality of persons or specifies the main subject A from a plurality of persons based on a user instruction.

  When the main subject A is automatically specified, the face detection unit 11 specifies the main subject A from a plurality of persons. For example, the face detection unit 11 identifies the face located closest to the main subject A among the detected face regions. This is because the person closest to the digital camera is generally likely to be the main subject for the user. At this time, the face detection unit 11 may set the person with the largest face size as the main subject A based on the size of the face, or based on the subject distance information acquired at the time of AF, A person may be determined. Further, the face detection unit 11 may specify a person who is in the position of the focus detection area during AF as the main subject A.

  Next, when the main subject A is specified based on the user's instruction, the control unit 10 receives an input of the position of the main subject A via the operation button 18 using a menu image. Then, the control unit 10 sets the person at the position designated by the user as the main subject A. At this time, the control unit 10 may perform AF using the position of the main subject A designated by the user as the focus detection area. Further, the control unit 10 may change the upper limit of the number of main subjects A based on user input, and may set a plurality of people as the main subject A.

  Even when the face detection unit 11 identifies the main subject A from an image including a plurality of persons in the digital camera 1 of the second embodiment, the same processing as described above may be performed.

1 is a block diagram illustrating a configuration of a digital camera 1 according to a first embodiment of the present invention. It is a figure which shows the luminance distribution of the luminescent point before and behind a blurring process. It is a figure which shows the example of a blurring shape. 6 is a flowchart illustrating an operation of the control unit 10 during imaging in the smile mode in the digital camera 1. It is a figure which shows the example of this image. It is a table | surface which shows the relationship between the main to-be-photographed object A or the background area | region B, and a blurring degree or a blurring shape. 4 is a flowchart illustrating an operation of the control unit 10 during imaging in the anger mode in the digital camera 1. 4 is a flowchart illustrating an operation of the control unit 10 during imaging in a sadness mode in the digital camera 1. It is a flowchart which shows operation | movement of the control part 10 at the time of reproduction | regeneration in the digital camera 1 of 2nd Embodiment. It is a figure which shows the example of the display part 14 at the time of reproduction | regeneration in the digital camera 1 of 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Image sensor, 6 ... Image processing part, 7 ... Compression / decompression processing part, 8 ... Analysis part, 10 ... Control part, 11 ... Face detection part, 12 ... Expression recognition part, 13 ... Subject extraction part , 14 ... display unit, 16 ... operation unit

Claims (16)

A subject extraction unit that extracts a main subject from an image and determines a range of a background area that does not include the main subject in the image;
An evaluation value calculation unit that evaluates the expression of the main subject and calculates an expression evaluation value indicating the degree of expression;
An image processing apparatus comprising: an image processing unit that performs a blurring process on the background area according to the facial expression evaluation value. The image processing apparatus according to claim 1.
The evaluation value calculation unit calculates an evaluation value indicating the degree of smile as the facial expression evaluation value,
The image processing device, wherein the degree of blur is increased as the facial expression evaluation value is higher. The image processing apparatus according to claim 2,
The image processing device, wherein the brightness of the background region is increased as the facial expression evaluation value is higher. In the image processing device according to any one of claims 1 to 3,
An analysis unit for analyzing the spatial frequency of the background region;
The image processing device, wherein the degree of blurring is increased as the number of high frequency components increases. The image processing apparatus according to any one of claims 1 to 4, wherein:
A ratio calculation unit for calculating a ratio of the face area of the main subject to the image;
The image processing device, wherein the blurring degree is increased when the ratio of the face area is equal to or greater than a threshold value. The image processing apparatus according to any one of claims 1 to 5, wherein:
A luminance calculation unit for calculating the luminance of the background region;
The image processing device, wherein the degree of blur is increased as the luminance is lower. The image processing apparatus according to any one of claims 1 to 6,
An estimation unit for estimating at least one of the age and sex of the main subject;
The image processing unit increases the degree of blurring when the estimation unit estimates that the age of the main subject is equal to or lower than a threshold value or the main subject is a woman. The image processing apparatus according to claim 7.
The image processing unit performs color correction to increase redness with respect to the face area of the main subject when the estimation unit estimates that the age of the main subject is equal to or less than a threshold value or the main subject is a woman. An image processing apparatus characterized by performing the processing. The image processing apparatus according to any one of claims 1 to 8,
The evaluation value calculation unit calculates an evaluation value indicating the degree of smile as the facial expression evaluation value,
The image processing apparatus, wherein the image processing unit changes a blur shape according to the expression evaluation value. In the image processing device according to any one of claims 1 to 9,
An estimation unit for estimating at least one of the age and sex of the main subject;
The image processing unit is configured to change a blur shape according to an estimation result of the estimation unit. The image processing apparatus according to claim 1.
The evaluation value calculation unit calculates an evaluation value indicating the degree of anger as the facial expression evaluation value,
The image processing device, wherein the image processing unit performs a blurring process only on an upper part of the head of the main subject when the facial expression evaluation value is equal to or greater than a threshold value. The image processing apparatus according to claim 1.
The evaluation value calculation unit calculates an evaluation value indicating the degree of sadness as the facial expression evaluation value,
The image processing apparatus, wherein the image processing unit performs color correction to make the color tone of the background region monotone or sepia when the facial expression evaluation value is equal to or greater than a threshold value. The image processing apparatus according to any one of claims 1 to 12,
A display unit for displaying an image of a predetermined facial expression among the images;
A selection unit that selects any one of the images displayed on the display unit;
The image processing unit performs blurring processing on the image selected by the selection unit. An imaging unit that acquires an image by imaging a main subject;
A digital camera comprising the image processing apparatus according to any one of claims 1 to 13,
The digital camera according to claim 1, wherein the image processing unit performs blurring processing on at least an image acquired by the imaging unit. The digital camera according to claim 14,
A digital camera, further comprising: an imaging control unit that controls the imaging unit according to a facial expression of the main subject. An image processing program for realizing image processing on image data to be processed by a computer,
A subject extraction step of extracting a main subject from the image and determining a range of a background area not including the main subject in the image;
An evaluation value calculating step for evaluating the expression of the main subject and calculating an expression evaluation value indicating the degree of the expression;
An image processing program comprising: an image processing step for performing a blurring process on the background area according to the facial expression evaluation value.

Images