JP2022140505A - Image processing apparatus, image processing method, and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method and a program therefor which decreases load of correction processing for faces in a plurality of captured images.

SOLUTION: An image processing apparatus has an image receiving unit 401, a face detecting unit 402, a feature point detecting unit 403, a representative pixel value determining unit 404, a first object position specifying unit 405, and a movement amount calculating unit 406. A second object position specifying unit 407 specifies a position of an object in each of images from the second image through the nth image based on the position of an object of the first image and a movement amount of a feature point between a first image and each of images from the second image through the nth image. A pixel value rewriting unit 409 rewrites, to a representative pixel value, pixel values of pixels of the objects in the images of objects in the images from the first image through the nth image. A through image generating unit 410 generates through images from rewritten images from the first image through the nth images.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art It is widely used to subject a person's face included in an image captured and recorded by a digital camera or the like to face correction processing such as removal of moles and whitening processing (for example, Patent Document 1). .

JP-A-11-73498

However, there is a problem in that the processing load is large when performing the face correction process for each of a plurality of captured images.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus, an image processing method, and a program that reduce the load of face correction processing on a plurality of captured images.

In order to achieve the object of the present invention, an image processing apparatus according to the present invention includes:
an image acquiring means for sequentially acquiring face images;
Detecting feature points in the face region of the face images in the first image acquired by the image acquisition means and the second image acquired after the acquisition of the first image, and detecting the positions of the feature points feature point position detection means for detecting
an object position identifying means for detecting an object within the face area of the face image in the first image obtained by the image obtaining means and identifying the position of the object;
comparison means for comparing the positions of the feature points of the first image and the positions of the feature points of the second image detected by the feature point position detection means;
Information for generating information for specifying the position of the object in the second image based on the comparison result by the comparing means and the position of the object in the first image specified by the object position specifying means a generating means;
rewriting means for rewriting a pixel value of the position of the object in the second image specified based on the information for specifying the position of the object in the second image generated by the information generating means;
characterized by comprising

According to the present invention, it is possible to provide an image processing device, an image processing method, and a program that reduce the load of face correction processing on a plurality of captured images.

1 is an external rear view of a digital camera according to Embodiment 1 of the present invention; FIG. 1 is a block diagram showing the configuration of a digital camera; FIG. 3 is a block diagram showing the configuration of an image processing unit; FIG. FIG. 2 is a diagram showing a human face object photographed by a digital camera; FIG. 3 is a diagram showing pixel values of a human face object captured by a digital camera, where (a) is the pixel value of area A, (b) is the pixel value of area B, and (c) is the pixel value of area C; show. FIG. 2 is a diagram showing the positions of objects and feature points in an image captured by a digital camera; FIG. 11 shows a table indicating the position of each object in the first image; FIG. 4 is a diagram showing changes in the positions of feature points and objects from the first image to the n-th image, where (a) is the first image, (b) is the second image, and (c) is the third image. , and (d) shows the respective changes in the nth image. FIG. 10 is a diagram showing a table showing changes in position of each feature point from the first image to the n-th image; It is a figure which shows the flowchart of a photography process. It is a figure which shows the flowchart of a photography process. FIG. 10 is a diagram showing a flowchart of photographing processing of the digital camera according to Embodiment 2 of the present invention; FIG. 4 is a diagram showing an image captured by a digital camera; FIG. 10 is a diagram showing a table showing personal data of multiple users;

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. INDUSTRIAL APPLICABILITY An image processing apparatus according to the present invention is used in an imaging apparatus having a function of performing correction processing for correcting the face of a person or the like on a captured face image. The imaging device has two modes, a skin-beautifying mode and a makeup mode. The skin-beautifying mode is a mode for removing moles, spots, wrinkles, rough skin, etc., and the makeup mode is a mode for adding a catch light to the eyes or giving the lips a so-called gloss that adjusts the brightness and color of the lips. . In the following embodiments, an imaging apparatus having both a beautiful skin mode and a makeup mode will be described.

(Embodiment 1)
FIG. 1 is a rear view of a digital camera 1 that is an example of an imaging device according to an embodiment of the invention. The digital camera 1 includes a camera body 2, a power switch 3 arranged on the top of the camera body 2, a release switch 4 arranged adjacent to the power switch 3 on the top of the camera body 2, and the camera body 2. A mode setting key 5 arranged on the rear surface of the camera, a reproduction key 6 for switching between a photographing mode and a reproduction mode, and a liquid crystal screen 7 for displaying an image. The release switch 4 is, for example, a two-stage push button that fixes the exposure and focus when pressed halfway, and captures an image when fully pressed. Although not shown, a flash and a photographing lens are arranged on the front surface of the camera body 2 .

In the digital camera 1, when the power switch 3 is pressed to turn on the power, the digital camera 1 enters a photographable state (photographing mode). In the photographing mode, a subject image incident through the photographing lens is output every frame (for example, 1/30 second), and the subject images are continuously displayed on the liquid crystal screen 7 as a through image. While viewing the through image, the user presses the release switch 4 to fix the image as a captured image and shoot it.

FIG. 2 is a diagram showing a schematic configuration of the digital camera 1. As shown in FIG. As shown in FIG. 2, the digital camera 1 includes a photographing lens 10, an image sensor 20, an image processing unit 30, an image processing unit 40, a DRAM (Dynamic Random Access Memory) 50, a flash memory 60, an image display unit 70, an operation input A section 80 and a CPU (Central Processing Unit) 90 are provided.

The photographing lens 10 is composed of a plurality of lenses (not shown), and includes a zoom lens and a focus lens. A lens driving circuit (not shown) is connected to the photographing lens 10 to drive the photographing lens 10 along the optical axis direction.

The imaging device 20 is specifically composed of a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and converts an object image that has passed through each lens of the photographing lens 10 into a two-dimensional image signal.

The imaging processing unit 30 includes a timing generator, a vertical driver, and a horizontal driver (not shown). The image signals are converted into dimensional image signals, and image frames corresponding to one screen are read from the imaging area of the imaging device 20 and output to the image processing unit 40 .

The image processing unit 40 converts the analog data of the frame image transferred from the imaging element 20 into digital data, performs white balance adjustment processing, color processing processing, etc., and outputs a luminance signal Y and color difference signals Cb and Cr. to generate

The DRAM 50 temporarily records image data output from the imaging processing unit 30 and data being processed by the image processing unit 40 .

The flash memory 60 is, for example, an electrically rewritable non-volatile memory, and stores a program for operating the digital camera 1 and realizing the functions of the digital camera 1, data used while the camera program is running, and the like. is pre-recorded.

The image display unit 70 includes a display processing unit 70a and a display unit 70b. The display processing unit 70a reads display data temporarily recorded in the DRAM 50 and controls display on the display unit 70b. Specifically, under the control of the CPU 90, the display processing unit 70a reads out the luminance signal Y and the color difference signals Cb and Cr from the DRAM 50, stores them in a VRAM (not shown), periodically reads them out from the VRAM, and displays the video signals. Output to unit 70b.

The display unit 70b displays an image captured by the imaging element 20 on the liquid crystal screen 7, which is a display screen, based on the video signal from the display processing unit 70a. When in the photographing mode, a through image periodically created by the image processing unit 40 is displayed, and when reproducing a recorded image, a captured image read out from the flash memory and decompressed is displayed.

The operation input unit 80 receives operation instructions of the digital camera such as shooting instructions, playback instructions, power-on instructions, and power-off instructions, setting operations related to the operation modes of the digital camera, such as shooting mode and playback mode, and shooting conditions setting operations by the user. It is for inputting various operations, and outputs an operation signal to the CPU 90 according to the operation input. Various functions are assigned to the operation input unit 80 and realized by the power switch 3, the release switch 4, the mode setting key 5, the playback key 6, and the like in FIG.

The CPU 90 receives input from the operation input section 80 and controls the overall processing of the digital camera 1 according to the program stored in the flash memory 60 .

A functional block diagram of the image processing section 40 is shown in FIG. The image processing unit 40 includes an image receiving unit 401, a face detection unit 402, a feature point detection unit 403, a representative pixel value determination unit 404, a first object position identification unit 405, a movement amount calculation unit 406, and a second object position identification unit. It includes a section 407 , an area adjustment section 408 , a pixel value rewriting section 409 , a through image creation section 410 , a first capture image creation section 411 , a second capture image creation section 412 , and a switching section 413 .

The image receiving unit 401 receives images from the first image to the n-th image (n is a positive integer) from the imaging processing unit 30 . In the present embodiment, the first image refers to an image that serves as a reference for specifying the position of the object in the second to n-th images. For the first image, a process of determining a representative pixel value and identifying the position of the object based on the representative pixel value, which will be described later, is performed in order to specify the position of the object from the second image to the n-th image. It is an image. For the second to n-th images, the process of determining the representative pixel value and specifying the position of the object is not performed. Also, the image receiving unit 401 and the imaging processing unit 30 form an image acquiring unit.

The face detection unit 402 detects whether or not there is a face area in the first to n-th images received by the image reception unit 401 . Specifically, by comparing and collating general human face feature data (feature data such as eyes, eyebrows, nose, mouth, ears, facial contour, etc.) stored in advance with the received image data , an area that matches face feature data equal to or greater than a predetermined value is detected as a face area. Alternatively, feature points may be extracted from each image, and the face area may be detected based on the feature points. Feature point detection is realized by processing using, for example, the Harris corner detection algorithm or AAA (Active Appearance Model). The face detection unit 402 forms face detection means.

A feature point detection unit 403 detects feature points in the face area. Feature points are features that can be detected within a person's unique features such as eyes, nose, mouth, facial contours, or other facial regions. If the feature points are used when the face detection unit 402 detects the face area, the feature points used at this time may be used as the feature points in the face area.

The detected feature points are located in XY coordinates, as shown in FIG. Specifically, the center point of the left eye is the position of the first feature point (X ₁ , Y ₁ ), and the center point of the right eye is the position of the second feature point (X ₂ , Y ₂ ). The head of the mouth is specified as the position of the third feature point (X ₃ , Y ₃ ), and the center point of the mouth is specified as the position of the fourth feature point (X ₄ , Y ₄ ). The value of the position of this feature point is stored in the table shown in FIG.

When the user selects the beautiful skin mode, the representative pixel value determining unit 404 measures the pixel values of the face area and determines the representative pixel value. Specifically, in this embodiment, a sampling area is set in the face area of the first image, and pixel values in the sampling area are sampled. As a sampling area, specify an area such as the bridge of the nose and cheeks where shadows are less likely to occur and where areas other than skin color such as eyes and nostrils are not included. Then, the representative pixel value of the pixels indicating the skin color of the detected face is determined. The shape of the sampling area is determined according to the position of the face to be sampled, such as rectangular, circular, or polygonal. In the present embodiment, the sampling area is specified for the nasal bridge of the face. The sampling area is a 16×16 pixel area, and 256 sampling points are arranged so as to overlap the pixels.

Then, the Y component, the Cb component, and the Cr component are obtained for each pixel in which the sampling points are arranged, and a histogram of each value is generated. An average value μ is calculated for the three generated histograms, and only pixels whose three values fall within a predetermined range centered on the average value μ are extracted and used as sampling pixels. For example, using the 2σ method, pixels existing within a range of ±2σ (where σ is the standard deviation) are taken as sampling pixels. Furthermore, a representative pixel value is determined using each average value of the Y component (luminance component), Cb component (color difference component), and Cr component (color difference component) of the sampled pixels.

When the makeup mode is selected, the representative pixel value is determined according to the extent of makeup stored in the flash memory 60 in advance. The makeup mode is executed by allowing the user to select the part of the face on which makeup is to be applied. For example, when the lips are selected as a part to be made up, lip gloss processing is performed. As the lip gloss processing, the degree of makeup with different brightness can be selected, and the representative pixel value is determined according to the degree of makeup selected by the user.

Based on the determined representative pixel value, the first object position specifying unit 405 compares the pixel value of the pixels in the face region of the first image with the representative pixel value, and finds a difference between the representative pixel value and the representative pixel value. Objects composed of pixels having pixel values greater than or equal to a predetermined value are detected. Here, the object is an object to be processed in the skin beautifying mode, such as a mole, blemish, wrinkles, or rough skin, or an object to be processed in the makeup mode, such as lips or eyes.

Specifically, the inside of the face area is divided into a plurality of predetermined lattice-like divided areas, and the representative value of the pixels is determined for each divided area. One divided area is defined by i×j pixels (i=j), and a divided area is defined by, for example, 6×6 pixels. An average value, a mode value, or a centroid value of pixel values is obtained for each divided area, and the obtained value is used as a representative value of the pixels of the divided area (hereinafter referred to as "area pixel value"). Then, the representative pixel value of the face area and the area pixel value of each divided area are compared to determine whether the difference between the pixel values is equal to or greater than a predetermined threshold.

When the skin-beautifying mode is selected, the area pixel value of the skin area is compared with the representative pixel value, and when the makeup mode is selected, the area where makeup is applied, for example, the area including the eyes and lips. A pixel value and a representative pixel value are compared.

The relationship between the representative pixel value of the face area, the area pixel value, and the predetermined threshold is expressed by Equation 1 below.
|Li−Lr|≧Lth (equation 1)
Here, Lr is the representative pixel value of the face area, Li is the area pixel value, and Lth is a predetermined threshold value. A region composed of pixels having a region pixel value that satisfies this formula is specified as an object region.

A method for specifying the position of an object will be specifically described using the beautiful skin mode as an example. If the representative pixel value is 200 and the threshold is 70, then the area in which pixels with pixel values equal to or higher than the threshold of 70 are arranged is identified as the object area. From Equation 1, |Li−200|≧70, and pixels exhibiting a pixel value of Li that satisfies this equation are identified.

FIG. 4 shows an example of an image having a first object 41, a second object 42, and a third object 43 on a photographed face. A first object 41 is in area A of the facial area, a second object 42 is in area B of the facial area, and a third object 43 is in area C of the facial area. An enlarged view of each region is also shown.

The pixel values of the pixels forming area A, area B, and area C are shown in FIGS. The pixel value shown here is the luminance Y. FIG. Values of other color elements may also be used as pixel values. Regions satisfying Equation 1 are shown as regions surrounded by thick lines in FIGS. 5(a) to 5(c). The area surrounded by thick lines in FIG. 5(a) is the first object 41, the area surrounded by thick lines in FIG. 5(b) is the second object 42, and the area surrounded by thick lines in FIG. 5(c). is the third object 43 .

The first object position specifying unit 405 determines the position of the central pixel of each of the first object 41, the second object 42, and the third object 43 as the position of the object. As shown in FIG. 6, in the XY coordinate system, the position of the first object 41 is (Xb1, Yb1), the position of the second object 42 is (Xb2, Yb2), and the position of the third object 43 is (Xb3 , Yb3), and stored in the table shown in FIG.

When the makeup mode is selected, the position of an object such as lips may be determined using data obtained when the face detection unit 402 detects the face area.

Based on the positions of the feature points of the first image and the positions of the feature points from the second image to the n-th image, the movement amount calculation unit 406 calculates the first image and the n-th image from the second image. Calculate the amount of movement of the image to each image. The movement amount calculation unit 406 detects the positions of the feature points from the position of the feature point of the first image to the n-th image for each of the plurality of feature points detected by the feature point detection unit 403 .

A method of calculating the amount of movement will be described with reference to FIG. 8, taking the movement of an arbitrary feature point as an example. In the figure, black circles indicate feature points, and areas indicated by a plurality of pixels are object areas. The feature point is located at the first position (Xr-1, Yr-1) in the XY coordinates in the first image 801, as shown in FIG. 8(a). The movement amount calculation unit 406 determines that the feature point is located at the second position (Xr-2, Yr-2) in the second image 802 as shown in FIG. 8(c) at the third position (Xr-3, Yr-3), and in the nth image 804, at the nth position (Xr-n, Yr-3) as shown in FIG. 8(d). n). Based on these coordinate positions, the amount of movement of the first feature point of each of the first image and the second to n-th images is calculated. That is, the amount of movement of the feature point from the first image to the second image is ((Xr-1)-(Xr-2), (Yr-1)-(Yr-2)), and (Δx1 , Δy1). Similarly, the movement amount of the feature point between the first image and the third image is obtained as (Δx2, Δy2). The amount of movement may be calculated based on the position of the feature point of the first image and the position of the feature point of each image, or the amount of movement between the images may be integrated to obtain the first image. may be calculated from the feature point to each image.

The movement amount calculation unit 406 executes the method of calculating the movement amount described above for each of the first to fourth feature points, and calculates the first to n-th images for each feature point. is obtained and stored in the table shown in FIG. Then, using the position data of the table shown in FIG. 9, the amount of movement from the first feature point to the fourth feature point is calculated.

The amount of movement may be obtained by detecting a plurality of feature points and tracking the movement of each feature point, or may be obtained by detecting one feature point and tracking the movement. Tracking the movement of one feature point requires less processing than tracking two or more feature points. On the other hand, when two or more feature points are tracked, the object can be identified by considering the orientation and tilt of the face. Further, when calculating the amount of movement, a reference frame may be determined, and motion vector information expressing the motion from the reference frame to the next frame as a vector may be obtained and used.

The second object position specifying unit 407 determines the position of the object in the first image, the amount of movement of the feature points between the first image and each of the second to n-th images, to identify the position of the object in each of the second to n-th images based on .

Specifically, the positions of the second to n-th images of the first object 41, the second object 42, and the third object 43 are specified based on the amount of movement of the feature points. For example, the position of the first object 41 in the second image is specified based on the movement amount (Δx1, Δy1) of the first feature point.

Further, by linking the positional relationship between the feature points and the object in the first image, the position of the object after movement can be specified accurately. As shown in FIG. 8A, the distance d between the feature point and the object can be calculated and the positions of the object and the feature point can be linked. For example, the distance between the position of one feature point of the contour of the face and the position of the mole, which is the object, is calculated, and the position of the object in the second and subsequent images is specified in consideration of this distance. be able to.

A pixel value rewriting unit 409 rewrites the pixel values of the pixels of the object from the first image to the n-th image with representative pixel values. For example, the pixel values of the first object 41 from the first image to the n-th image are rewritten with 200, which is the representative pixel value. The rewriting of the pixel values may be performed by individually rewriting the pixel values of the pixels that make up the first object 41, or by creating a partial image with the area of the object that makes up the first object 41 as the representative pixel value. , this partial image may be synthesized with the original image.

The pixel value rewriting unit 409 does not rewrite the pixel values of the pixels forming the object when the movement amount calculated by the movement amount calculation unit 406 is equal to or less than the first predetermined movement amount. For example, if the amount of movement of the object from the first image to the second image is zero, or if the amount of movement is such that the movement cannot be recognized in the displayed image, the pixel values of the pixels of the destination object are not rewritten. do not have. Therefore, when the pixels constituting the object of the first image are rewritten with the representative pixel values, if there is no movement of the object from the first image to the second image, the object of the second image is It is possible to use an image that has already been rewritten to the representative pixel value in one image. Therefore, the processing load can be reduced. Further, when the movement amount calculated by the movement amount calculation unit 406 is equal to or greater than the second predetermined movement amount, the image reception unit 401 receives the first image. For example, if the amount of movement is too large, it is difficult to track the object, so the processing returns to receiving the first image.

A region adjusting unit 408 adjusts the region of the object to be rewritten by the pixel value rewriting unit 409 . Rewriting only the pixels located in the same area as the object area may result in an unnatural appearance if there is a difference in pixel value between the rewritten pixels and adjacent pixels in the through image or captured image. may become.

In order to eliminate the unnatural appearance due to the difference in pixel value between the rewritten pixels and the adjacent pixels, for example, the first object position specifying unit 405 may add Based on this, the pixel values of the pixels surrounding the specified object are stored, and when there is a difference of a predetermined value or more between the surrounding pixel value and the representative pixel value, the pixel value rewriting area is set wider than the object area.

For example, when the pixel values of a mole object are rewritten to representative pixel values, if there is a predetermined difference between the pixel values around the mole and the representative pixel values, the rewritten portion is highlighted. will be Therefore, the area of pixel values to be rewritten is adjusted to be wider than the area of the object so that the difference between the pixel values around the mole and the pixel value of the representative pixel value gradually changes, and the area of the representative pixel value object Pixel values around the mole are gradually changed toward the outside.

Also, the area adjustment unit 408 can make the range of pixels to be rewritten reflecting the user's intention larger or smaller than the area of the object. In order to determine the range of pixels to be rewritten, a selection unit (not shown) for selecting the area to be rewritten is provided to allow the user to determine the area range in which the pixels are to be rewritten. The region adjustment unit 408 determines the range of pixels to be rewritten according to the user's selection signal received from the selection unit. For example, when performing gloss processing, the user selects a lip range to be subjected to gloss processing, and the pixel value rewriting unit 409 rewrites the pixel values of the selected lip range to representative pixel values.

The through image creating unit 410 creates a through image by rewriting the pixel values of the objects from the first image to the n-th image with representative pixel values. The created through image is temporarily recorded in the DRAM 50 and continuously displayed on the liquid crystal screen 7 by the display section 70b. The through image is created by thinning out and reading out the image data of the subject image formed on the imaging device 20 . The through image is updated at a predetermined frame rate (here, 30 fps) and displayed as a continuously displayed moving image.

Note that the feature point detection unit 403, the representative pixel value determination unit 404, the first object position identification unit 405, and the second object position identification unit 407 form correction information generation means. Also, the first object position specifying unit 405 and the second object position specifying unit 407 form object detection means. The feature point detection unit 403 corresponds to feature point detection means, and the representative pixel value determination unit 404 corresponds to representative pixel value determination means.

The area adjusting unit 408, the pixel value rewriting unit 409, and the movement amount calculating unit 406 form processing means. Further, the movement amount calculation unit 406 corresponds to movement amount calculation means and includes comparison means. The pixel value rewriting unit 409, through image creating unit 410, and area adjusting unit 408 form object processing means. The pixel value rewriting unit 409 corresponds to pixel value rewriting means.

A first captured image creation unit 411 acquires a photographed image when the user presses the release switch 4 and creates a first captured image. Specifically, an image receiving unit 401 receives a captured image, a face detection unit 402 detects a face area of the captured image, a feature point detection unit 403 detects feature points of the face area, and determines a representative pixel value. A representative pixel value of the face area is determined by the unit 404, and the position of the object in the captured image is specified by the first object position specifying unit 405. FIG. Then, the first capture image creation unit 411 creates a capture image in which the pixel values of the object regions of the second and subsequent images are rewritten with the representative pixel values. The first captured image creating section 411 corresponds to first captured image creating means.

A second captured image creation unit 412 acquires a photographed image when the user presses the release switch 4 and creates a second captured image. Specifically, the captured image is received by the image receiving unit 401, and the representative pixel value and the position of the object used in the through image created from the image received when the release switch 4 is pressed are added to the received captured image. Based on, the pixel values of the pixels in the object area are rewritten with the representative pixel values. The second captured image creating section 411 corresponds to the second captured image creating means 412 .

The switching unit 413 uses the image receiving unit 401, the face detection unit 402, the feature point detection unit 403, the representative pixel value determination unit 404, the first object position identification unit 405, and the second object position identification unit 407 to select the through image. Execution processing of the first captured image creation unit 411 and the second capture image creation unit 412 is switched according to the processing load at the time of creation. Specifically, when the processing load when creating a through image is heavy, the second captured image creation unit 412 is switched to be executed, and when the processing when creating a through image is light, Switching is performed so that the first capture image creation unit 411 is executed. The switching unit 413 corresponds to switching means.

Next, the operation of the digital camera 1 when the beautiful skin mode is selected will be described with reference to the flowchart of FIG.

When the power switch 3 of the digital camera 1 is pressed, the camera enters the shooting mode, and frame images are continuously acquired via the image pickup device 20 and the image pickup processing section 30 . The user operates the mode setting key 5 to select the beautiful skin mode. Signals of frame images obtained in succession are received by the image receiving unit 401 of the image processing unit 40 . (Step S101). When the signal of the first image is received, face detection section 402 detects a face area in the first image (step S102). The feature point detection unit 403 extracts feature points such as the mouth, nose, and eyes from the detected face area (step S103).

Next, the representative pixel value determining unit 404 obtains the representative pixel value of the pixels in the face area of the first image from the sampled pixels (step S104). Then, the first object position identifying unit 405 identifies the area of the object in the first image and identifies the position of the object (step S105). The pixel value rewriting unit 409 rewrites the pixel values of the pixels in the object area with the representative pixel values (step S106), and the through image creating unit 410 creates a through image of the first image (step S107).

Next, the image receiving unit 401 receives the second image (step S109), and the face detection unit 402 detects the face area of the second image (step S110). The feature point detection unit 403 detects feature points of the second image (step S111), and the movement amount calculation unit 406 calculates the amount of movement of the feature points from the first image to the second image (step S112). ). Whether or not the movement amount is too large is determined by determining whether or not the movement amount is equal to or greater than a second predetermined value (step S113). The object position specifying unit 407 specifies the position of the object in the second image based on the amount of movement from the first image to the second image (step S114). If the amount of movement is greater than or equal to the second predetermined value (step S113: YES), the process returns to step S101 for receiving the first image.

Next, the pixel value rewriting unit 409 rewrites the pixel values of the pixels in the area of the object in the second image with the representative pixel values (step S115), and the through image creating unit 410 rewrites the pixels to create a through image of the second image. is created (step S116). Note that if the amount of movement is equal to or less than the first predetermined value, the pixels are not rewritten.

The processing from step S109 to step S116 is repeated from the second image to the n-th image.

Next, it is determined whether or not the release switch 4 is ON (step S117). If the release switch 4 is ON (step S117; YES), the process proceeds to A, and if the release switch 4 is not ON (step S117; NO), the process ends.

When the release switch 4 is turned on, the process proceeds to A, and as shown in FIG. 11, a photographed image is received (step S118). When the release switch 4 is half-pressed, the CPU 90 sets the focus adjustment state in which the focus is fixed. Then, when the release switch 4 is fully depressed, the exposure is controlled and the image data pixels are read out.

Then, the image data of the captured image at the timing when the release switch 4 is fully pressed is received by the image receiving section 401 (step S118). Next, the face detection unit 402 detects the face area of the captured image (step S119), and the feature point detection unit 403 detects feature points within the face area (step S120).

Next, the representative pixel value determining unit 404 determines representative pixel values of pixels in the face area (step S121), and the first object position identifying unit 405 identifies the position of the object in the captured image (step S122). ). Then, the pixel value rewriting unit 409 rewrites the pixel values of the pixels in the area of the object in the captured image with the representative pixel values (step S123), and the first captured image creating unit 411 or the second captured image creating unit 412 creates a captured image (step S124). The created captured image is displayed on the liquid crystal screen 7 by the display section 70b. The created captured image is recorded in the storage medium, and the process ends.

According to the present embodiment, the amount of movement of the feature point detected in the first image from the first image to each image after the second image, and the position of the object specified in the first image. , it is possible to specify the position of the object in each image after the second image. Therefore, since it is not necessary to individually detect the position of the object in each image after the second image, the load of image processing can be reduced.

According to the present embodiment, since the through image and the captured image are created by specifying the position of the object from the first image to the n-th image, the through image and the captured image can be used to represent the corrected object. No big difference in appearance. Therefore, the user does not feel uncomfortable between the through image and the captured image.

According to the present embodiment, it is possible to specify a plurality of objects in the first image and to specify the positions of the objects after movement. can be processed together.

According to the present embodiment, digital camera 1 includes first captured image creation unit 411, second capture image creation unit 412, and switching unit 413 for switching between these creation units. Processing can be switched in relation to the processing load, and processing efficiency can be improved.

According to the present embodiment, since the area adjustment unit 408 is provided, the corrected image of the object can be adjusted to look natural or desired by the user.

(Embodiment 2)
In Embodiment 1, the first image is obtained and the positions of feature points and objects are searched. The user may be identified by matching the captured face with the stored face region data. Then, the feature points of the user and the position of the object may be registered as personal registration data, and the through image may be created using the personal registration data.

The DRAM 50 or a detachable recording medium (not shown) contains image data for identifying the user's face area photographed in advance, and a table shown in FIG. The value is stored in association with the user as personal registration data.

FIG. 12 shows a flowchart of image processing when personal registration data is used.

Image processing starts when the user turns on the power switch 3 . A first image is acquired by the imaging element 20, and the image receiving unit 401 receives an image signal of the first image (step S201). A face area in the first image is detected by the face detection unit 402 (step S202), and personal registration data stored in a recording medium or the like is searched by a search unit (not shown) based on the detected face area. (Step S203), it is determined whether the detected face area and the saved face area match (Step S204). If there is matching face area data (step S204; YES), the individual is specified, and the pixel values of the pixels of the object are calculated based on the specified user's feature point positions, object positions, and representative pixel values. The representative pixel value is rewritten to create a first through image (step S205). If there is no matching face area data (step S204; NO), the first image is newly received.

Next, the position of the object from the second image to the n-th image is specified in the same manner as in the first embodiment, and the object area is rewritten with the representative pixel value. Specifically, the image receiving unit 401 receives the image of the second image (step S206), the face detection unit 402 detects the face area of the second image (step S207), and the feature point detection unit 403 Feature points are detected from the image (step S208). The movement amount calculation unit 406 calculates the movement amount of the feature point (step S209). Whether or not the amount of movement is too large is determined by determining whether or not the amount of movement is equal to or greater than a second predetermined value (step S210). , the object position specifying unit 407 specifies the position of the object in the second image (step S211). If the amount of movement is greater than or equal to the second predetermined value (step S210: YES), the process returns to the process of receiving the first image. The pixel rewriting unit 409 rewrites the specified object area with the representative pixel value (step S212), and the through image creating unit 410 creates a through image (step S213). This process is repeated from the second image to the nth image. When the release switch 4 is pressed by the user (step S214; YES), the process proceeds to the capture image creation process shown in FIG. If the release switch 4 is not pressed by the user (step S214; NO), the process is terminated.

The user's personal registration data to be stored is not limited to one user, and data of a plurality of users can also be stored. As shown in FIG. 13, even when three persons are displayed on one screen, the same processing as when personal data of one user is registered can be performed. For example, as shown in the table of FIG. 14, for user X, user Y, and user Z, by storing the personal registration data of each user, the second image to the n-th image for each of the three persons can be obtained. object can be located.

13(a) shows the display screen of the first image 131, and FIG. 13(b) shows the display screen of the second image 132. FIG. Even if the three persons X, Y, Z in the first image 131 move within the second image 132, each person can be identified based on their personal registration data. Therefore, even if the positions of the persons are changed, the positions of the persons can be identified, and the positions of the objects corresponding to the movements of the persons can be easily identified.

According to this embodiment, by storing the face region, the position of the object, and the position of the feature point in advance, the face region, the position of the object, and the position of the feature point in the first image can be easily specified. can reduce the processing load.

In Embodiments 1 and 2, the amount of movement is calculated based on the coordinate position of the feature point in the XY coordinates. In the present invention, additional information can be added to such a method of calculating the amount of movement.

Specifically, information such as the distance between facial feature points, the orientation of the face, and the tilt of the face may be considered. For example, the distance between the eyes may be calculated from the positional information of both eyes, which is the feature point, the enlargement or reduction ratio of the image may be obtained according to the distance, and the enlargement or reduction ratio may be reflected in the movement amount. . Alternatively, the inclination of the face may be obtained from the angle formed by the line connecting the center points of both eyes and the horizontal line and reflected in the amount of movement. Furthermore, when only one eye is detected, it is possible to determine that the face is turned sideways and not calculate the amount of movement.

By using the additional information in this way, the position of the object can be accurately specified, and the processing burden can be reduced.

In Embodiments 1 and 2, it has been described that the first image is a reference image for specifying the position of the object in the second image and subsequent images. Which image should be the first image is determined based on whether the image is an image that facilitates the detection of the face area and the detection of the feature points. For example, as the first image, it is possible to use an image taken when the subject is facing the front, an image taken at predetermined intervals, or the like.

In Embodiments 1 and 2, it was explained that the switching unit 413 switches between the first captured image creating unit 411 and the second captured image creating unit 412 according to the load. , and the second capture image creation unit 412 . By applying only the first captured image creation unit 411, the resolution of the image displayed in the captured image can be improved. The processing load can be reduced by applying only the second captured image creation unit 411 . In addition, regardless of which capture image creation unit is applied, it is possible to reduce the difference in appearance from the through image.

Although Embodiments 1 and 2 describe the imaging apparatus having both the skin-beautifying mode and the makeup mode, the imaging apparatus may have either mode.

Although the skin-beautifying mode and makeup mode have been described in Embodiments 1 and 2, other modes may be provided. For example, it has an effect mode. The effect mode is a mode in which a star mark, a heart mark, or the like is given, or a star mark, a heart mark, or the like whose color or size changes with time.

Although a digital camera has been described as an example in this embodiment, the image processing apparatus of the present invention is not limited to a digital camera. It can also be applied to image processing devices such as mobile phones, information terminals, and smart phones.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to such specific embodiments, and the present invention includes the invention described in the claims and their equivalents. be The invention described in the original claims of the present application is appended below.

(Appendix)
(Appendix 1)
an image acquiring means for sequentially acquiring images;
face detection means for detecting a face area in the image acquired by the image acquisition means;
a correction information generation means for generating correction information for correcting the first image acquired by the image acquisition means;
processing means for processing the correction information based on a second image different from the first image acquired by the image acquisition means;
An image processing device comprising:

(Appendix 2)
The correction information generating means is
feature point detection means for detecting feature points in the face area;
an object detection means for detecting an object within the face area;
The image processing device according to appendix 1.

(Appendix 3)
The processing means is
comparison means for comparing the feature points of the first image and the feature points of the second image detected by the feature point detection means;
an object processing means for processing the object detected by the object detection means according to the comparison result;
The image processing device according to appendix 2.

(Appendix 4)
The comparison means is
the first image based on the position of the feature point of the first image detected by the face detection means and the position of the feature point in each of the second to n-th images; , each image from the second image to the n-th image, and a movement amount calculation means for calculating the movement amount of the feature point between the images,
The image processing apparatus according to appendix 3.

(Appendix 5)
The correction information generating means is
a representative pixel value determining means for determining a representative pixel value representing pixel values of pixels in the face region of the first image acquired by the image acquiring means;
The image processing device according to appendix 4.

(Appendix 6)
The object detection means is
A pixel value of each pixel in the face area of the first image is compared with the representative pixel value, and an object composed of pixels having a pixel value different from the representative pixel value by a predetermined value or more is detected. do,
The image processing apparatus according to appendix 5.

(Appendix 7)
The object detection means is
a first object locating means for locating the object in the first image;
7. The image processing apparatus according to appendix 5 or 6.

(Appendix 8)
The object detection means is
based on the position of the object in the first image and the amount of movement of the feature points between the first image and each of the second to n-th images, a second object locating means for locating the object in each of the nth images from the images;
The image processing device according to appendix 7.

(Appendix 9)
The processing means is
pixel value rewriting means for rewriting pixel values of pixels of the object in each of the first image to the n-th image to the representative pixel value;
The image processing apparatus according to appendix 8.

(Appendix 10)
The processing means is
a through-the-lens image creating means for creating a through-the-lens image including the object processed by the processing means from the images sequentially acquired by the image acquiring means;
The image processing apparatus according to appendix 3.

(Appendix 11)
further comprising a first captured image creating means for creating a first captured image when a photographing instruction is input;
The image acquisition means acquires a photographed image obtained in response to the photographing instruction,
The face detection means detects a face area in the captured image,
The correction information generation means generates correction information for correcting the captured image acquired by the image acquisition means,
The processing means processes the correction information based on an image different from the photographed image acquired by the image acquisition means,
The first captured image creating means creates the first captured image of the captured image processed by the processing means.
11. The image processing device according to appendix 10.

(Appendix 12)
a second captured image creating means for creating a second captured image when a shooting instruction is input;
The second capture image creation means is
creating the second captured image based on the correction information used when creating the through image when the shooting instruction is input;
11. The image processing apparatus according to appendix 10.

(Appendix 13)
a first captured image creating means for creating a first captured image when a photographing instruction is input;
a second captured image creating means for creating a second captured image when a photographing instruction is input;
further comprising switching means for switching between processes executed by the first capture image creation means and the second capture image creation means;
The switching means selects the first capture image creation means, the first capture image creation means, and the switching execution processing of the second capture image creation means,
When switched to the first capture image creation means,
The image obtaining means receives a photographed image obtained in response to the photographing instruction,
The face detection means detects a face area in the captured image,
The correction information generation means generates correction information for correcting the captured image acquired by the image acquisition means,
The processing means processes the correction information based on an image different from the photographed image acquired by the image acquisition means,
When switched to the second capture image creation means,
The second capture image creation means is
creating the second captured image based on the correction information used when creating the through image when the shooting instruction is input;
11. The image processing device according to appendix 10.

(Appendix 14)
The pixel value rewriting means changes the pixel value of the pixel of the object in the destination image when the amount of movement of the feature point calculated by the movement amount calculating means is equal to or less than a first predetermined amount. , do not rewrite,
9. The image processing apparatus according to appendix 9.

(Appendix 15)
The object processing means creates a partial image corresponding to the object, arranges the partial image at the position of the object, and processes the partial image.
The image processing device according to appendix 3.

(Appendix 16)
The first object position identifying means detects a plurality of objects and identifies the position of each of the plurality of objects;
The second object position specifying means is configured to position each of the plurality of objects in the first image and between the first image and each of the second to n-th images. and specifying the position of each of the plurality of objects in each of the second image to the n-th image based on the amount of movement of the feature point.
The image processing device according to appendix 8 or 9.

(Appendix 17)
The processing means is
Further comprising area adjusting means for adjusting a range of pixels whose pixel values are to be rewritten by said pixel value rewriting means to be larger or smaller than the area of pixels constituting said object,
The image processing device according to appendix 9.

(Appendix 18)
The image acquisition means acquires the first image when the movement amount of the feature point calculated by the movement amount calculation means is equal to or greater than a second predetermined amount.
The image processing device according to appendix 4.

(Appendix 19)
obtaining a first image;
detecting facial regions in the first image;
generating correction information for correcting the first image;
and processing the correction information based on a second image that is different from the first image.
Image processing method.

(Appendix 20)
to the computer,
obtain a first image;
detecting a face region in the first image;
generating correction information for correcting the first image;
processing the correction information based on a second image different from the first image;
A program for executing a process.

DESCRIPTION OF SYMBOLS 1... Digital camera, 2... Camera main body, 3... Power switch, 4... Release switch, 5... Mode selection key, 6... Reproduction button, 7... Liquid crystal screen, 10... Lens, 20... Imaging device, 30... Imaging processing unit , 40... Image processing unit 41... First object 42... Second object 43... Third object 50... DRAM 60... Flash memory 70... Image display unit 70a... Display processing unit 70b Display unit 80 Operation input unit 90 CPU 401 Image reception unit 402 Face detection unit 403 Feature point detection unit 404 Representative pixel value determination unit 405 First object position identification unit , 406... Movement amount calculation unit 407... Second object position specification unit 408... Area adjustment unit 409... Pixel value rewriting unit 410... Through image creation unit 411... First capture image creation unit 412... Second capture image creation unit 413 Switching unit 131, 801 First image 132, 802 Second image 803 Third image 804 nth image

In order to achieve the object of the present invention, an image processing apparatus according to the present invention includes:
an image acquiring means for sequentially acquiring images;
Detecting feature points in regions of the first image acquired by the image acquisition means and a second image acquired after the acquisition of the first image, and detecting positions of the feature points feature point position detection means for detecting
an object position identifying means for detecting an object within a region of the image with respect to the first image obtained by the image obtaining means and identifying the position of the object;
comparison means for comparing the positions of the feature points of the first image and the positions of the feature points of the second image detected by the feature point position detection means;
Information for generating information for specifying the position of the object in the second image based on the comparison result by the comparing means and the position of the object in the first image specified by the object position specifying means a generating means ;
Prepare .

Claims (6)

an image acquiring means for sequentially acquiring face images;
Detecting feature points in the face region of the face images in the first image acquired by the image acquisition means and the second image acquired after the acquisition of the first image, and detecting the positions of the feature points feature point position detection means for detecting
an object position identifying means for detecting an object within the face area of the face image in the first image obtained by the image obtaining means and identifying the position of the object;
comparison means for comparing the positions of the feature points of the first image and the positions of the feature points of the second image detected by the feature point position detection means;
Information for generating information for specifying the position of the object in the second image based on the comparison result by the comparing means and the position of the object in the first image specified by the object position specifying means a generating means;
rewriting means for rewriting a pixel value of the position of the object in the second image specified based on the information for specifying the position of the object in the second image generated by the information generating means;
An image processing device comprising: The comparison means determines the feature points of the first image based on the amount of movement of the feature points between the positions of the feature points of the first image and the positions of the feature points of the second image. 2. The image processing apparatus according to claim 1, wherein the feature point of the second image is compared with the feature point of the second image. The object position specifying means generates information for specifying the position of the object included in the face image based on a representative pixel value representing pixel values of pixels in the face region of the first image. 3. The image processing apparatus according to claim 1, wherein: 4. The image processing apparatus according to any one of claims 1 to 3, further comprising through image creating means for creating a through image of a face image including said object whose pixel values have been rewritten by said rewriting means. . An image processing method executed by an image processing device,
an image acquisition step of sequentially acquiring face images;
detecting feature points in the face region of the face image in the first image acquired in the image acquisition step and the second image acquired after the acquisition of the first image; a feature point position detection step for detecting positions;
an object position specifying step of detecting an object in the face region of the face image from the first image acquired in the image acquiring step and specifying the position of the object;
a comparison step of comparing the positions of the feature points of the first image and the positions of the feature points of the second image detected in the feature point position detection step;
generating information for specifying the position of the object in the second image based on the comparison result of the comparing step and the position of the object in the first image specified in the object position specifying step; an information generating step;
a rewriting step of rewriting the pixel value of the position of the object in the second image, which is specified based on the information for specifying the position of the object in the second image generated in the information generating step; ,
An image processing method characterized by comprising: The computer of the image processing device,
image acquisition means for sequentially acquiring face images;
Detecting feature points in the face region of the face images in the first image acquired by the image acquisition means and the second image acquired after the acquisition of the first image, and detecting the positions of the feature points Feature point position detection means for detecting
Object position identifying means for detecting an object in the face area of the face image from the first image obtained by the image obtaining means and identifying the position of the object;
comparison means for comparing the positions of the feature points of the first image and the positions of the feature points of the second image detected by the feature point position detection means;
Information for generating information for specifying the position of the object in the second image based on the comparison result by the comparing means and the position of the object in the first image specified by the object position specifying means generating means,
rewriting means for rewriting the pixel value of the position of the object in the second image specified based on the information for specifying the position of the object in the second image generated by the information generating means;
A program characterized by functioning as

Images