JP2011197999A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a picturesque image corresponding to expression content of a human image in a photographed image.SOLUTION: A face image included in the photographed image stored in an image storage unit is detected (S103), and further, the expression of the detected face image is determined (S104). Thereafter, according to the determination result of the expression of the face image, a background image where the face image is removed from the photographed image is converted into a picturesque image (S105), the converted picturesque image and the face image are combined, and the combined picturesque image is displayed on a display unit (S106).

Description

  The present invention relates to an image processing apparatus and an image processing method for creating a painting-like image based on a photographed image.

  An apparatus that converts a photographed image into a painting-like image is known (see, for example, Patent Document 1 below).

JP 2004-213598 A

  However, in the case of such a conventional apparatus, there is a problem that the photographed image is only converted into a painting-like image by a predetermined image processing, and it is not interesting.

  The present invention has been made in view of these conventional problems, and an object thereof is to obtain an image processing apparatus and an image processing method capable of reflecting the expression content of a photographed subject in an image such as a background image. And

In order to solve the above-described problem, an image processing apparatus according to the invention of claim 1 is provided:
In an image processing apparatus including an image storage unit that stores captured images,
Subject image detection means for detecting a subject image having a facial expression included in the captured image stored in the image storage unit;
Facial expression discrimination means for discriminating the facial expression of the subject image detected by the subject image detection means;
In response to the facial expression detected by the facial expression detection means, image conversion means for converting a background image obtained by removing the subject image from the photographed image into a painting-like background image;
A painting-like image creation unit that synthesizes the subject image stored in the image storage unit and the painting-like background image converted by the image conversion unit, and creates the combined painting-like image;
It is characterized by having.

  An image processing apparatus according to a second aspect of the present invention is the image processing apparatus according to the first aspect, wherein the facial expression detection means discriminates a facial expression based on a mouth image in the subject image.

  An image processing apparatus according to a third aspect of the present invention is the image processing apparatus according to the second aspect, wherein the facial expression detection means discriminates a facial expression based on the shape of the mouth image.

  The image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to the first aspect, wherein the facial expression detection means discriminates a facial expression based on a combination of the shape of the mouth image and the eye image in the subject image. It is characterized by doing.

An image processing method according to a fifth aspect of the present invention comprises:
In an image processing method used in an image processing apparatus including an image storage unit that stores a captured image,
A subject image detection step for detecting a subject image having a facial expression included in the captured image stored in the image storage unit;
A facial expression determination step for determining the facial expression of the subject image detected by the subject image detection step;
Corresponding to the facial expression detected in this facial expression detection step, a painting-like image conversion step for converting a background image obtained by removing the subject image from the photographed image into a painting-like background image;
An image synthesis step of synthesizing the subject image stored in the image storage unit and the painting-like background image converted by the image conversion step;
It is characterized by having.

  An image processing method according to a sixth aspect of the present invention is the image processing method according to the fifth aspect, wherein the facial expression detection step discriminates a facial expression based on a mouth image in the subject image.

  The image processing method according to a seventh aspect of the present invention is the image processing method according to the sixth aspect, wherein the facial expression detection step discriminates the type of facial expression based on the shape of the mouth image.

  The image processing method according to an eighth aspect of the present invention is the image processing method according to the fifth aspect, wherein the facial expression detection means discriminates a facial expression based on a combination of the shape of the mouth image and the eye image in the subject image. It is characterized by doing.

An image processing apparatus according to the invention of claim 9 is provided.
An image storage unit that stores a captured image, a subject image detection unit that detects a subject image that has a facial expression included in the captured image stored in the image storage unit, and displays the image In an image processing apparatus having a display unit for
Facial expression determination means for determining the facial expression of the subject image detected by the subject image detection unit;
Corresponding to the facial expression content detected by the facial expression detection means, either a background image obtained by removing the subject image from the photographed image or the photographed image is converted into a painting-like image, and the converted image Painting-like image display control means for displaying a painting-like image on the display unit;
It is characterized by having.

  An image processing apparatus according to a tenth aspect of the present invention is the image processing apparatus according to the ninth aspect, wherein the painting-like image display control means corresponds to the facial expression content detected by the facial expression detection means. Special image display control means for adding a special image, converting the added special image into a painting-like image, and displaying the converted painting-like image on the display unit. It is characterized by that.

An image processing method according to the invention of claim 11 is provided.
An image storage unit that stores a captured image, a subject image detection unit that detects a subject image that has a facial expression included in the captured image stored in the image storage unit, and displays the image In an image processing method used in an image processing apparatus including a display unit for
A facial expression determination step for determining a facial expression of the subject image detected by the subject image detection unit;
Corresponding to the facial expression content detected by the facial expression detection step, the background image obtained by removing the subject image from the photographed image and the photographed image are converted into a painting-like image, and the converted image A painting-like image display control step for displaying a painting-like image on the display unit;
It is characterized by having.

  An image processing method according to a twelfth aspect of the present invention is the image processing method according to the eleventh aspect, wherein the painting-like image display control step corresponds to the facial expression content detected by the facial expression detection step. And a special image display control step of adding a special image, converting the added special image into a painting-like image, and displaying the converted painting-like image on the display unit. It is characterized by being.

  According to the present invention, it is possible to reliably and easily create a painting-like image in which the expression content of a photographed subject image is reflected in an image such as a background image.

  In addition, according to the present invention, a subject image (for example, a mouth image or the like) having a facial expression included in a captured image stored in the image storage unit is detected, and the detected subject image is detected. After creating a painting-like background image corresponding to the above, the created painting-like background image and the subject image are synthesized, so that an interesting image can be created.

  Furthermore, according to the present invention, it is possible to display a composite image in which the subject image and the painting-like background image are combined on the display unit.

It is a figure which shows the external appearance of the digital camera and display apparatus which concern on one embodiment of this invention. It is a block block diagram which shows the outline of an electrical structure with the digital camera and a display apparatus. It is a memory block diagram of the painting style conversion reference ROM. It is a figure which shows an example of a mouth image in a picked-up image. It is a memory block diagram of RAM. It is a memory block diagram of discrimination | determination result RAM. It is a flowchart which shows the painting-like image creation process procedure in one embodiment of this invention. It is a flowchart which shows the specific procedure in the facial expression discrimination | determination process of FIG. It is a flowchart which shows the specific procedure in the painting style conversion process of FIG. It is a figure which shows an example of the picked-up image of the person who is laughing. It is a figure which shows an example of the painting style image after performing the painting style image creation process according to the picked-up image of FIG. It is a figure which shows an example of the picked-up image of an angry person. It is a figure which shows an example of the painting style image after performing the painting style image creation process according to the picked-up image of FIG. (A) is a figure which shows the picked-up image of the person laughing similar to the picked-up image of FIG. 10, (b) is after performing the painting-like image creation process according to the picked-up image of this laughing person It is a figure which shows an example of another painting-like image.

Hereinafter, embodiments of the present invention will be described.
FIG. 1 is a rear view showing the appearance of a digital camera 1 and a display device 100 according to an embodiment of the present invention.
On the back of the digital camera 1, a menu key 2, a mode dial 3, a display unit 4, a cursor key 5, a SET key 6, and the like are provided. A shutter key 8 and a power button 9 are provided on the upper surface. Further, on the front side, an imaging unit 10 including an imaging lens described later is provided. A captured image or the like captured by the digital camera 1 is stored in a built-in RAM 22 or memory card 24 described later.
On the other hand, on the surface of the display device 100, a display unit 4 for displaying a photographed image or the like stored in the attached memory card 24 is provided. Display device 100 is controlled by remote control device 101.

FIG. 2 is a block configuration diagram showing an outline of an electrical configuration of the digital camera 1 and the display device 100.
The digital camera 1 includes an imaging unit 10 and a CPU 11. The former imaging unit 10 includes a CCD. The latter CPU 11 compresses / decompresses an image captured by the CCD, and generates an image file in a format that conforms to a predetermined format, for example, a unified standard DCF (Design Rule for Camera File system) or JPEG (Joint Photographic Expert Group). An image processing function for playback is provided.

  An optical image of the subject is formed on the light receiving surface of the CCD constituting the imaging unit 10 through the imaging lens, the focus lens, and the diaphragm. The focus lens is held by a driving mechanism such as an AF motor, and a focusing signal that moves back and forth on the optical axis when a driving signal output from the AF driver is supplied to the driving mechanism by a control signal from the CPU 11. Do. The diaphragm is driven by a drive signal generated by the diaphragm drive unit based on a control signal from the CPU 11 and adjusts the amount of light of the subject image incident on the CCD.

  Further, a TG (Timing Generator) that generates a timing signal is connected to the CPU 11, and a V driver (vertical driver) drives the CCD based on the timing signal generated by this TG. An analog imaging signal corresponding to the brightness of the signal is output and sent to the unit circuit.

  The unit circuit includes a CDS that holds an imaging signal output from the CCD, a gain adjustment amplifier (AGC) that is an analog amplifier to which the imaging signal is supplied from the CDS, and an analog imaging that is amplified and adjusted by the gain adjustment amplifier. It comprises an A / D converter (AD) that converts the signal into a digital signal. The output signal of the CCD is sampled with the black level matched here and sent to the CPU 11 as a digital signal. The gain adjustment amplifier (AGC) amplifies the imaging signal supplied from the CDS with an amplification factor corresponding to the ISO sensitivity.

  In addition to the input unit 21 including the key group including the shutter key 8, the display unit 4, the RAM 22, the ROM 23, and the memory card 24 are connected to the CPU 11.

  The RAM 22 is a working memory, and the memory card 24 is an image recording memory. An imaging signal, which is a digital signal sent from the CCD to the CPU 11, is temporarily stored in the RAM 22, subjected to various image processing by the CPU 11, and finally recorded as compressed image data in the memory card 24. Is done. The memory card 24 is of a card type that is detachable from the camera body.

  A captured image or the like recorded on the memory card 24 is read by the CPU 11 as necessary, and is converted into a digital video signal or an analog video signal through processes such as expansion processing, addition of a luminance signal and a color signal, and a display unit. 4 is displayed. In addition, on the display unit 4, an image periodically captured in the recording mode is displayed as a through image.

The ROM 23 is a program ROM in which various operation programs necessary for control of each unit and data processing in the CPU 11 are recorded. The ROM 23 stores various data such as a program diagram indicating a combination of ISO sensitivity and shutter speed corresponding to an appropriate exposure value at the time of photographing together with the program.
The ROM 23 also stores painting style conversion reference data for converting the background image into a painting image format corresponding to the facial expression. Specifically, since facial expressions, particularly facial expressions, appear in the mouth, this painting-like conversion reference data includes an expression storage unit 231 for storing facial expression data relating to facial expressions, and a destination picture relating to a destination picture image conversion format. It is stored in a conversion format storage unit 232 for storing the image conversion format. The former facial expression storage unit 231 stores the type and degree of the mouth shape, and the latter conversion format storage unit 232 stores the type and degree of the picture image format. The type and degree of the shape of the mouth and the type and degree of the picture image format are stored in association with each other by 1 to 10 of “number” in association with each other.

Here, in this embodiment, examples of the shape of the mouth include a laughing mouth, an angry mouth, a soliciting mouth and a fun mouth, and the degree is, for example, a laughing mouth, a laughing mouth belonging to the category of the laughing mouth, A laughing mouth, a laughing mouth and others. An angry mouth, a solicitation mouth, and a pleasant mouth each include mouths of different degrees belonging to the category. In FIG. 3, the symbols A to Z described in the column of the degree next to the column of the shape of the mouth represent the degree of difference. The type and degree of the shape of the mouth are, for example, any one of the ratio of the vertical size α and the horizontal size β of the mouth shown in FIG. It is determined in advance in consideration of some combinations. Of course, other determination elements may be set.
In the present embodiment, the types of painting image formats include watercolor-tone images, ink-tone images, oil-tone images, and gothic images, and changes in the degree include changes in brightness and degree of arrangement. In FIG. 3, the symbols A to Z described in the column next to the format column indicate the degree of change in the picture image format.
Note that the mouth level (A to Z) and the picture image format level (A to Z) preferably correspond one to one. In addition, it is preferable to use a bright painting image format when the mouth is a laughing mouth or a pleasant mouth, and to a dark painting image format when the mouth is an angry mouth or a solicitation mouth.

  The CPU 11 performs AE control by controlling the charge accumulation time of the CCD, which is the shutter speed, the degree of opening of the diaphragm, and the gain of the gain adjustment amplifier (AGC) of the unit circuit, and drives the focus lens according to a predetermined program. AF (auto focus) control and AWB (auto white balance) control are performed. The memory card 24 stores captured images and the like.

In the RAM 22, a picture-like conversion data area 221 shown in FIG. 5 is provided. The painting-like conversion data area 221 includes a first storage unit 222, a second storage unit 223, a third storage unit 224, a fourth storage unit 227, and a fifth storage unit 229.
Among these, the first storage unit 222 stores captured images. The second storage unit 223 stores a person image, which is a subject image having a facial expression, whose contour is extracted from the captured image, and the remaining background image. Further, the third storage unit 224 stores a mouth image obtained by extracting a contour from a person image. The fourth storage unit 227 stores a painting-like background image obtained by converting the background image in a predetermined painting image format and the coordinate position of the painting-like background image. In addition, the fifth storage unit 229 stores a composite image after the person image and the painting-like background image are combined.
The position coordinates in the fourth storage unit 224 are the coordinates of all dots in the entire area of the painting-style background image, the coordinates of only the painting-style background image, and the coordinates of a plurality of points in the painting-style background image. Any coordinates may be used as long as the coordinates can specify the position.

  In addition, as shown in FIG. 6, the RAM 22 is provided with a data area 226 for storing a facial expression discrimination result and a conversion destination picture image discrimination result in association with each other. This data area 226 stores the shooting number, the type / degree of mouth shape, and the type / degree of destination picture format.

  On the other hand, the display device 100 has the same configuration as that of the digital camera 1 except for the photographing unit 10. Therefore, in the display device 1, the same components as those of the photographing unit 10 are denoted by the same reference numerals, and description thereof is omitted. The display device 100 is controlled by a remote control operation of the remote control device 101.

  In the present embodiment having the above configuration, the digital camera 1 executes the painting-like image creation process shown in the flowchart of FIG. 7 by the CPU 11 executing the process according to the program stored in the ROM 23.

  That is, when the user carries the digital camera 1 and, for example, presses the shutter key 8 under a scene where the subject is the background of Mt. Fuji and the front is a person, the CPU 11 starts the photographing process, and the photographing unit Thus, as shown in FIG. 10, the captured image 6 including the Mt. Fuji image 6a and the person image 6b is acquired (step S101).

  Next, the CPU 11 stores the acquired captured image 6 in the first storage unit 222 in the RAM 22 and executes a captured image display process to display the captured image 6 on the display unit 4 (step S102). ). As a result, the captured image 6 shown in FIG. 10 is displayed on the display unit 4.

  Subsequently, the CPU 11 detects a person image existing in the captured image 6 (step S103). This person image detection is performed using a known person image detection technique. When the person image detection is completed, the CPU 11 stores the person image 6 b and the background image 6 a obtained by removing the person image 6 b from the photographed image in the second storage unit 223 in the RAM 22.

Thereafter, the CPU 11 performs facial expression determination processing of the person image (step S104).
Specifically, this facial expression discrimination process is performed according to the flowchart of the facial expression discrimination process shown in FIG.
That is, the CPU 11 detects a mouth image as shown in FIG. 4 from the person image (step S1041). This mouth image detection is performed using a known mouth image detection technique. Next, the CPU 11 stores the mouth image obtained by the mouth image detection in the third storage unit 224 of the RAM 23 (step S1042), and the ratio between the mouth vertical size α and the horizontal size β shown in FIG. The type and degree of the shape of the mouth are determined in consideration of the area ratio and the rise and fall of the mouth angle (step S1043). The area ratio of the teeth to the mouth is, for example, the ratio of the tooth area (area shown by hatching) to the area of the entire mouth image.
Then, the CPU 11 stores the determination result in the RAM 23 (step S1404), and ends the facial expression determination process.

Next, returning to the flowchart of FIG. 7, the CPU 11 performs a painting-style conversion process based on the type and degree of the conversion destination image format corresponding to the type and degree of the shape of the mouth indicating the discrimination result by the facial expression discrimination process described above ( Step S105).
Specifically, this painting-style conversion process is performed according to a flowchart of the painting-style conversion process shown in FIG.
That is, the CPU 11 refers to the facial expression storage unit 231 storing the painting style conversion reference data in the ROM 23 according to the discrimination result by the facial expression discrimination process described above, searches for the type and degree of the shape of the mouth, and then converts With reference to the conversion format storage unit 232 for storing the destination picture image conversion format related to the destination picture image conversion format, the type and degree of the destination picture image format corresponding to the discrimination result of the type and degree of the mouth shape are converted. Read from the format storage unit 232 (step S1051).
Next, the CPU 11 stores the type and degree of the read destination picture image format in the discrimination result RAM 22 of FIG. 6 (step S1052).
Next, the background image read from the RAM 22 is converted in accordance with the type and degree of the conversion destination picture image format (step S1053). Then, the CPU 11 stores in the fourth storage unit 227 the picture-like background image obtained by the conversion process and the coordinate position of the picture-like background image (step S1054).
Subsequently, the CPU 11 synthesizes the picture-like background image and the person image read from the second storage unit 223 in accordance with each coordinate position stored corresponding to the image (step S1055). Then, the CPU 11 stores the synthesized image as a painting-like image in the fifth storage unit 229 of the RAM 22 (step S1056), and ends the painting-like conversion process.

  Subsequently, returning to the flowchart of FIG. 7 again, the CPU 11 displays the synthesized painting-like image on the display unit 4 (step S106), and ends the painting-like image creation process.

FIG. 10 shows a captured image 6 including a person image 6b with a laughing mouth. FIG. 11 shows a painting-like background image 6A after painting-like image creation processing such as bright colors and bright brush touches according to the expression content of the person image 6b with a laughing mouth. Note that the painting-like image 6A shown in FIG. 11 includes a sun T that is not included in the photographed image 6 as a background image in which the painting-like background image 6A having a bright color as a whole is further brightened. Good.
On the other hand, FIG. 12 shows a photographed image 6 including an angry person image 6b. FIG. 13 shows a painting-like background image 6B after a painting-like image creation process such as dark color or dark brush touching according to the expression content of the angry person image 6b. In the painting-like image 6B shown in FIG. 13, the photographed image 6 may include a suspicious cloud K, and the painting-like background image 6B having a dark color as a whole may be a darker background image.

  According to the embodiment as described above, the facial expression of the person image included in the stored photographed image is detected, and the painting-like background image corresponding to the facial expression content is created. Since it is possible to synthesize the created painting-like background image and the person image, an interesting painting-like image including a painting-like background image corresponding to the facial expression content can be obtained quickly and reliably. .

  Further, if the displayed picture-like image is stored in the memory card 24 and the stored memory card 24 is attached to the display device 100, the composite image can be viewed on the display unit 4 having a large screen.

As mentioned above, although embodiment of this invention was described, it cannot be overemphasized that this invention is not limited to this embodiment.
For example, after or before the composite image is displayed on the screen of the display unit 4, the type and degree of the conversion destination image format to be used are displayed, and if necessary, by input from the input unit 21. The CPU 11 may be configured so that the CPU 11 can execute the processing from step S105 to step S109 according to the type and degree of the input conversion destination image format. In this way, the user can obtain a desired image.

  In the above-described embodiment, the facial expression is determined based on the mouth image. However, the facial expression may be determined in consideration of the image of the eye where the facial expression is relatively likely to appear. In this case, it is considered whether the eyes are hanging up or hanging down.

  Furthermore, in the above-described embodiment, based on the facial expression content of the person image in the photographed image, a picture-like background image obtained by converting only the background image in the photographed image into a painting style, and a person image that has not been converted into a painting style, And the synthesized picture image is displayed. As shown in FIG. 14, the person image 6b and the background image are displayed on the basis of the expression content of the person image 6b in the photographed image 6. The entire photographed image 6 including 6a may be converted into a painting-like image, and the converted painting-like person image 6C and the painting-like image including the painting-like background image 6A may be displayed together. In this case, since the entire photographed image 6 including the picture-like person image 6C and the picture-like background image 6A is converted into the picture-like, a picture-like image rich in unexpectedness can be obtained.

In the present embodiment, the case where the present invention is applied to a digital camera has been described. However, the present invention is not limited to a digital camera and may be applied to other electronic devices such as a mobile phone.
In this embodiment, the facial expression of a person image is determined as a subject image having a facial expression. However, an animal image, a portrait image, a character, an avatar image (an avatar image), a virtual creature image, etc. It is possible to discriminate the facial expression of the subject image having the facial expression, convert the background image in the photographed image into a painting style based on the discrimination result of the facial expression, and display the converted painting style image Good.

1 Digital Camera 4 Display Unit 10 Imaging Unit 11 CPU
24 memory card

Claims (12)

In an image processing apparatus including an image storage unit that stores captured images,
Subject image detection means for detecting a subject image having a facial expression included in the captured image stored in the image storage unit;
Facial expression discrimination means for discriminating the facial expression of the subject image detected by the subject image detection means;
In response to the facial expression detected by the facial expression detection means, image conversion means for converting a background image obtained by removing the subject image from the photographed image into a painting-like background image;
A painting-like image creation unit that synthesizes the subject image stored in the image storage unit and the painting-like background image converted by the image conversion unit, and creates the combined painting-like image;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the expression detection unit determines an expression based on a mouth image in the subject image.   The image processing apparatus according to claim 2, wherein the facial expression detection unit discriminates a facial expression based on a shape of the mouth image.   The image processing apparatus according to claim 1, wherein the expression detection unit determines an expression based on a combination of a shape of a mouth image and an eye image in the subject image. In an image processing method used in an image processing apparatus including an image storage unit that stores a captured image,
A subject image detection step for detecting a subject image having a facial expression included in the captured image stored in the image storage unit;
A facial expression determination step for determining the facial expression of the subject image detected by the subject image detection step;
Corresponding to the facial expression detected in this facial expression detection step, a painting-like image conversion step for converting a background image obtained by removing the subject image from the photographed image into a painting-like background image;
An image synthesis step of synthesizing the subject image stored in the image storage unit and the painting-like background image converted by the image conversion step;
An image processing method comprising:   6. The image processing method according to claim 5, wherein the facial expression detection step discriminates a facial expression based on a mouth image in the subject image.   The image processing method according to claim 6, wherein the expression detection step determines the type of expression based on the shape of the mouth image.   6. The image processing method according to claim 5, wherein the facial expression detection means discriminates a facial expression based on a combination of a shape of a mouth image and an eye image in the subject image. An image storage unit that stores a captured image, a subject image detection unit that detects a subject image that has a facial expression included in the captured image stored in the image storage unit, and displays the image In an image processing apparatus having a display unit for
Facial expression determination means for determining the facial expression of the subject image detected by the subject image detection unit;
Corresponding to the facial expression content detected by the facial expression detection means, either a background image obtained by removing the subject image from the photographed image or the photographed image is converted into a painting-like image, and the converted image Painting-like image display control means for displaying a painting-like image on the display unit;
An image processing apparatus comprising:   The painting-like image display control means adds a special image to the photographed image corresponding to the facial expression content detected by the facial expression detection means, and also adds the added special image to the painting-like image. The image processing apparatus according to claim 9, further comprising special image display control means for converting the image into an image and displaying the converted picture image on the display unit. An image storage unit that stores a captured image, a subject image detection unit that detects a subject image that has a facial expression included in the captured image stored in the image storage unit, and displays the image In an image processing method used in an image processing apparatus including a display unit for
A facial expression determination step for determining a facial expression of the subject image detected by the subject image detection unit;
Corresponding to the facial expression content detected by the facial expression detection step, the background image obtained by removing the subject image from the photographed image and the photographed image are converted into a painting-like image, and the converted image A painting-like image display control step for displaying a painting-like image on the display unit;
An image processing method comprising:   The painting-like image display control step adds a special image to the photographed image corresponding to the facial expression content detected by the facial expression detection step, and also adds the added special image to the painting-like image. The image processing method according to claim 11, further comprising a special image display control step of converting the image into an image and displaying the converted picture image on the display unit.

Images