JP2015041865A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform blurring only of the background region by discriminating the region accurately, when discriminating the subject region and background region from a plurality of captured images of different focus, by using the edge information.SOLUTION: An image processing method includes a blurring step for blurring a captured image, and a synthesizing step for synthesizing a captured image subjected to blurring and a captured image not subjected to blurring, according to the region discrimination results obtained in a region discrimination step, and generating a composite image where the background is blurred. In the region discrimination step, after performing region discrimination with the size of a first region, region discrimination is performed for a region, where the region discrimination results of the first region discrimination are not the subject region nor the background region, with the size of a second region smaller than the size of the first region.

Description

  The present invention relates to a technique for discriminating a subject area and a background area of a photographed image.

  Many compact digital cameras have been developed that have a function to artificially blur the background in image processing. As an example of the technique, a plurality of pieces of image data are photographed by changing the in-focus position, the subject area and the background area are discriminated based on the difference information, and the image data of the background area is subjected to blurring processing. A method of combining with image data is mentioned.

  Patent Document 1 discloses a technique for discriminating a subject area and a background area by comparing the power of a high-frequency component between an image focused on a subject and an image focused on infinite distance.

JP 2003-283902 A

  In Patent Document 1, it is possible to determine the subject area and the background area by comparing the power of the high-frequency component between the image focused on the subject and the image focused on the infinite distance.

  However, it is effective for a subject having a high frequency component, but when the subject and the background are low-contrast scenes, the high frequency component is small.

  Accordingly, an object of the present invention is to provide an image processing apparatus capable of discriminating an appropriate subject area and background area when generating an image in which the background of the subject is blurred.

  In order to solve the above-described problem, an image processing apparatus according to the present invention compares an acquisition unit that acquires a plurality of photographed images with different focal positions and a signal of the plurality of photographed images for a plurality of regions of the photographed image. Region determining means for determining an area for determining a subject area and a background area, a blur processing means for performing blur processing on the photographed image, a photographed image subjected to blur processing by the blur processing means, and the blur A synthesized image that is combined with a photographed image that has not been processed in accordance with a region discrimination result obtained by the region discriminating unit and generates a blurred composite image of the background region, the region discriminating unit comprising: After performing the region determination with the size of the first region, the region determination result of the first region determination is the previous region that is neither the subject region nor the background region. And performing the area determination by the size of the size smaller than the second region of the first region.

  According to the image processing apparatus of the present invention, it is possible to determine an appropriate subject area and background area when generating an image in which the background of the subject is blurred.

It is a figure which shows schematic structure of the imaging device of this embodiment. It is a figure which shows an example of a structure of the digital signal processing part 109 in FIG. It is a flowchart figure of imaging | photography operation | movement of the imaging device of this embodiment. FIG. 4 is a flowchart of a shooting subroutine in FIG. 3. It is area | region discrimination explanatory drawing in FIG.

(First embodiment)
FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus including an image processing apparatus to which an embodiment of the present invention is applied.

  The image pickup apparatus of the present embodiment includes an optical system 101 and a focus lens 102, and picks up an image of a subject by forming incident light on the solid-state image pickup device. The optical system control unit 103 controls exposure (shutter speed and aperture value), zoom, optical blur correction, and the like of the optical system 101, and the focus lens drive control unit 104 controls the focus lens 102. The focus lens drive control unit 104 drives the focus lens 102 by using a contrast signal of image data, a signal from a distance measuring sensor provided separately from the imaging surface of the solid-state image sensor 105 and the solid-state image sensor 105, and AF (autofocus). ) Control. The solid-state imaging device 105 converts an optical image formed through an RGB color filter on an imaging surface such as a Bayer array into an electrical signal. The imaging system control unit 106 is a control system for driving the solid-state imaging device 105. The analog signal processing unit 107 performs processing such as clamping and gaining on the output of the solid-state image sensor 105. An analog / digital (A / D) conversion unit 108 performs digital conversion of an analog signal and outputs it as image data. In the present embodiment, the imaging unit including the optical system 101 to the A / D conversion unit 108 is used, and the imaging apparatus records a still image by recording image data output from the imaging unit alone or continuously in a plurality of frames. And video can be taken. In addition, the image processing apparatus of the present invention does not necessarily have an imaging function, and may be acquired by reading image data stored in advance.

  The digital signal processing unit 109 performs development processing and compression processing on the A / D converted digital image data to generate output image data. The development processing performed by the digital signal processing unit 109 includes, for example, demosaicing processing, white balance processing, shading correction processing, color space conversion processing, and gamma processing. The internal storage unit 110 temporarily stores image data when the digital signal processing unit 109 generates output image data, image data for an electronic viewfinder (hereinafter referred to as EVF) operation for displaying a live image, and the like. To do.

  The interface (I / F) unit 111 is connected to an external recording device for finally storing the generated output image data. The image display unit 112 has a display function of EVF operation, and displays various setting information such as shooting conditions of the imaging apparatus in addition to a live image.

  The system control unit 113 controls the entire system such as the optical system control unit 103, the focus lens drive control unit 104, and the imaging system control unit 106 according to a flowchart described later. Reference numeral 114 denotes a switch for performing a photographing standby operation (hereinafter referred to as SW1), reference numeral 115 denotes a photographing switch for performing photographing after the operation of SW1 (hereinafter referred to as SW2), and reference numeral 116 denotes a main switch for turning on the system. (Hereinafter referred to as main SW).

  FIG. 2 is a diagram showing an example of the configuration of the digital signal processing unit 109 in FIG.

  The input image signal is input to the background blurred image generation unit 203 through a luminance signal processing unit 201 that performs luminance signal processing and a color signal processing unit 202 that performs color signal processing. The internal storage unit 110 is used for temporary storage during background blur processing. The image signal compressed and output by the image compression unit 204 is stored in an external recording device via the interface unit 111.

  Further, the background blurred image generation unit 203 includes an edge detection unit 205, an edge integral value calculation unit 206, an edge integral value evaluation unit 207, an area map generation unit 208, a blur processing unit 209, and an image composition unit 210. The background blur created western part 203 temporarily stores the history information of various processes in the internal storage unit 110 as necessary.

  FIG. 3 is a flowchart showing the photographing operation of the image pickup apparatus to which the embodiment of the present invention is applied.

  First, in step S301, the state of the main SW for turning on the system is detected. If it is ON, the process proceeds to step S302. In step S302, the remaining capacity of the external recording device via the interface unit 111 is checked. If the remaining capacity is 0, the process proceeds to step S303, and if not, the process proceeds to step S304. In step S303, a warning is given that the remaining capacity is 0, and the process returns to step S301. The warning may be displayed on the image display unit 112, a warning sound may be output from a sound output unit (not shown), or both may be performed.

  In step S304, an EVF operation for displaying the live image output from the imaging unit on the image display unit 112 is performed. In step S305, the state of the switch SW1 is checked. If it is ON, the process proceeds to step S307, and if not, the process proceeds to step S306. Here, the function of SW1 is to perform a shooting standby operation such as AF or AE (automatic exposure control). In step S306, the state of the main SW is checked. If it is ON, the process proceeds to step S304, and if not, the process proceeds to step S301. In step S307, AE processing is performed via the optical system control unit 103, and in step S308, AF processing is performed via the focus lens drive control unit 104.

  Step S309 after the shooting standby operation is completed is a preparation state before starting the shooting operation by turning on SW2, and the live image output from the imaging unit is displayed on the image display unit 112 and waits for the photographer's SW2 operation. .

  In step S310, the state of SW2 is checked. If ON, the process proceeds to step S312; otherwise, the process proceeds to step S311. Here, the function of SW2 is to start a photographing operation after the operation of SW1. In step S311, the state of SW1 is checked. If it is ON, the process returns to step S309; otherwise, the process returns to step S304. In step S312, a photographing operation is performed according to the flowchart of FIG. In step S313, the remaining capacity of the external recording device via the interface unit 111 is checked. If the remaining capacity is 0, the process proceeds to step S303, and if not, the process proceeds to step S314. In step S314, the state of SW2 is checked. If it is not ON, the process proceeds to step S311.

  FIG. 4 is a flowchart of a shooting subroutine in the flowchart of FIG.

  This flowchart shows a flow of processing from capturing two images for background blur processing to generating a background blur processing image by the background blur generation unit 203.

  In this embodiment, two images with different focal positions, an image focused on a main subject (hereinafter referred to as a main image) and a background detection image taken by moving the focus lens in a direction in which the background is in focus. Take a picture of. However, the number of images to be photographed with different focal positions is not limited to this, and may be a plurality of images.

  First, in step S401, the main image is taken at the focus lens position focused on the main subject set in the AF process of step S308 in the flowchart of FIG.

  In step S402, the focus lens is moved to capture the background detection image. The amount of movement of the focus lens is preset for each shooting condition such as aperture and focal length, and the focus lens can be detected from the sharpness change resulting from the difference in focus between the main image and the background detection image. Amount.

  In step S403, a background detection image is photographed at the focus lens position moved in step S402.

  With the imaging processing so far, two captured images, the main image focused on the main subject and the background detection image focused more on the background, could be obtained. The following is a flow of background blur processing by the background blur generation unit 203.

  In step S404, the edge detection unit 205 performs edge detection on the luminance signals of the two captured images of the main image and the background detection image, and obtains edge images (edge information), respectively. Here, it is assumed that the edge of the luminance signal is obtained by subjecting a captured image to band pass filter processing. Also, other methods may be used as the edge extraction method.

  In step S405, the edge integral value calculation unit 206 and the edge integral value evaluation unit 207 perform region discrimination on the edge image obtained in step S404 according to the flowchart of FIG.

  In step S406, the area map generation unit 208 generates a cutout map that can determine the subject area and the background area based on the area determination result in step S405. When generating a cutout map, a low-pass filter may be applied to make a discontinuous portion at the boundary between the background region and the main subject region inconspicuous. In the present embodiment, the cutout map is generated in an image format in which the subject area is 255, the background area is 0, the boundary area is an intermediate value, and the composition ratio for each pixel is expressed by an 8-bit pixel value. However, the map format is not limited to this.

  In step S407, the blur processing unit 209 performs blur processing based on the cutout map on the main image to generate a background blurred image. For the blurring process, a more favorable background blurring process can be performed, for example, by selecting a filter reflecting a lens characteristic with a beautiful blur.

  In step S408, the image composition unit 210 performs image composition from the main image focused on the subject (without blurring processing) and the background blurred image obtained by performing blurring processing on the main image based on the cut-out map.

Specifically, the image composition unit 210 performs image composition of the main image IMG1 focused on the subject and the background blurred image IMG2 based on α [i, j] (0 ≦ α ≦ 255) obtained from the pixel value of the cut-out map. To generate a composite image B. The image composition unit 210 calculates each pixel B [i, j] of the composite image B using the following equation (1).
B [i, j] = IMG1 [i, j] * α [i, j] / 255 + IMG2 [i, j] * (1-α) / 255 Expression (1)

  FIG. 5 is a diagram for explaining the area discrimination processing in step S405 of the flowchart of FIG. FIG. 5A is a flowchart of region discrimination processing, FIG. 5B is a flowchart of region evaluation processing in the flowchart of region discrimination processing in FIG. 5A, and FIG. It is explanatory drawing.

  In the flowchart of FIG. 5A, a cycle of dividing into a plurality of areas, evaluating each area, and correcting the isolated evaluation result area is performed a plurality of times. First, after performing a relatively coarse area division (area division with the size of the first area), fine area division (the size of the first area) is performed on the intermediate area and the low contrast area that are neither the subject area nor the background area. (Dividing the area by the size of the second area smaller than that).

  In step S501, the entire image is divided into a plurality of regions. The image to which the division is applied is each edge image of the main image and the background detection image obtained in the flowchart step S404 in FIG. 4, and the region division is associated with the two images.

  In step S502, the edge integral value calculation unit 206 and the edge integral value evaluation unit 207 perform region evaluation for each region divided and set in step S501 in accordance with the flowchart of FIG. As the region evaluation result, an evaluation result of any one of the subject region, the background region, the boundary region, and the low contrast region is obtained according to the edge integral value and the subject ratio.

  In step S503, for the region where the region evaluation result of step S502 is isolated from the evaluation result of the surrounding region, the isolated region is removed by correction that matches (reflects) the evaluation result to the surrounding evaluation result.

  In step S504, region division is performed with a size smaller than the size set in step S501 for the divided regions whose boundary evaluation results after the isolated region removal are boundary regions and low-contrast regions. Similar to step S501, the image to which the division is applied is each edge image of the main image and the background detection image obtained in the flowchart step S404 in FIG. 4, and the area division is common to the two images.

  In step S505, region evaluation is performed for each region set in step S504 in the same manner as in step S502. In step S506, as in step S503, for the region where the region evaluation result in step S505 is isolated from the evaluation result of the surrounding region, the isolated region is removed by correction that matches (reflects) the evaluation result to the surrounding evaluation result.

  As described above, the region evaluation result by the coarse region division and the region evaluation result by the fine region division are respectively the final region discrimination result for the region finely divided again in step S504 and the region not finely divided again. Become.

  Next, the flowchart of the area evaluation process in FIG. 5B will be described. Here, description will be made as a flow for obtaining a region evaluation for one divided region of interest. However, in step S502 or step S505 in the flowchart of FIG. 5A, the area evaluation subroutine of FIG. 5B is executed for each divided area to be subjected to area evaluation.

  In step S511, the edge integration value calculation unit 206 performs edge integration of the divided region of interest on two images, the edge image of the main image and the edge image of the background detection image obtained in step S404 of the flowchart of FIG. Calculate the value. Hereinafter, steps S512 to S518 are performed for each region.

  In step S512, the edge integral value evaluation unit 207 adds the edge integral value obtained from the edge image of the main image in the region of interest and the integral value obtained from the edge image of the background detection image to a predetermined threshold value. Check if this is the case. If the total edge integral value is greater than or equal to the predetermined threshold value, the process proceeds to step S514; otherwise, the process proceeds to step S513.

  In step S513, since the total edge integral value of the region of interest is less than the predetermined threshold, the region evaluation result of the region of interest is set as a low contrast region.

  In step S514, the value of the subject ratio obtained by dividing the edge integral value obtained from the edge image of the main image by the integral value obtained from the edge image of the background detection image in the edge integral value evaluation unit 207 is greater than or equal to a predetermined subject threshold value. Find out. If the subject ratio is greater than or equal to the predetermined subject threshold, the process proceeds to step S515, and if not, the process proceeds to step S516. The subject ratio should be calculated to be high in a subject area with different focus and different edge components in the main image and the background detection image, and to be detected low in the background area.

  In step S515, since the subject ratio is an area that is equal to or larger than the predetermined subject threshold value, the area evaluation result of the target area is set as the subject area. As described above, the cutout map is generated in an image format in which the subject area is 255, the background area is 0, the boundary area is an intermediate value thereof, and the composition ratio for each pixel is expressed by a pixel value of 8 bits. 255 is assigned to the area recognized as the subject area.

  In step S516, the edge integral value evaluation unit 207 checks whether the subject ratio value is less than the predetermined background threshold value. If it is less than the predetermined background threshold value, the process proceeds to step S517, and if not, the process proceeds to step S518.

  In step S517, since the value of the subject ratio is less than the predetermined background threshold, the region evaluation result of the region of interest is set as the background region, and zero is assigned in the extraction map.

  In step S518, since it is difficult to determine whether the subject area or the background area from the value of the subject ratio, an intermediate value from 0 to 255 is assigned in the cut-out map using the area evaluation result of the target area as a boundary area.

  When the region of interest is occupied by the subject or the background, the subject ratio value can be evaluated based on the predetermined subject threshold or the predetermined background threshold. However, if it is neither, for example, it may be a region of interest that occupies half the area at the boundary between the subject and the background, so the region evaluation result is a boundary region.

  In this way, the region evaluation is performed for each region of interest. At the time of region evaluation by the rough divided region in the flowchart step S502 in FIG. 5A, and at the region evaluation by the fine divided region in the flowchart step S505 in FIG. Then, each predetermined threshold value is set differently. In addition to the reason for the difference in size of the region of interest, at the time of region evaluation using a rough divided region, the region evaluation is performed again using the next fine divided region even if it is determined as a boundary region. However, this is because it is necessary to reduce the determination of the boundary region as much as possible because the region is not re-evaluated in the region determination based on the fine divided regions. When the region evaluation result by the fine divided region is the boundary region determination, the region evaluation result may be corrected in consideration of the edge integral value in the rough divided region.

  An explanatory diagram of area division in FIG. 5C will be described. c-0 is an overall view of an image to be subjected to background blurring processing, and c-1 to c-3 are diagrams illustrating an example of region division processing in region discrimination and region evaluation. In addition, c-1 to c-3 are diagrams showing only the region near the right ear of the main subject shown in c-0 for the sake of simplicity.

  c-1 is a diagram showing the result of area evaluation by rough area division, c-2 is a figure showing the area evaluation result by fine area division of the boundary area of c-1, and c-3 shows the effect of this embodiment. It is the figure which showed the area | region evaluation result by the fine area | region division from the beginning as an example to compare easily.

  When fine region discrimination is performed from the beginning as in c-3, earrings with white skin or glossy metal surfaces are likely to be judged as low contrast and are subject regions. Nevertheless, there is a high risk of errors being handled incorrectly with the background area. On the other hand, as shown in the present embodiment, when the area is discriminated with c-2 and the divided area after c-1 as two stages, the edge integral value is obtained with a large value in the coarse divided area. In addition, there is an advantage that even when a low contrast determination is made, it is possible to correct a simple region result like the above-described isolated region removal. In a scene where the boundary area is relatively simple, such as a general human scene, it is possible to discriminate between the subject and the background without blurring a part of the human face as the background area, for example. .

  In addition, since the boundary region is the second region discrimination with fine divided regions, the resolution necessary for the cut-out map is also ensured. And since the area of the boundary region where the region determination is performed again is limited, the increase in the calculation time required for the region determination processing can be reduced compared with the case where the entire screen is determined again.

  In the flowcharts of step S503 and step S506 in FIG. 5A of the present embodiment, the region evaluation result based on the edge integral value is corrected only by the relationship with the peripheral region, but the present invention is not limited to this. For example, the color information of each divided area is used as auxiliary information for area determination, such as performing grouping processing of divided areas using the color signal of each divided area and correcting the area evaluation result of the same group. Also good. Further, face detection information (subject detection information) as a result of known face detection by pattern matching or the like may be used as auxiliary information for obtaining the main subject region.

  As described above, according to the present embodiment, it is possible to provide an image processing apparatus that performs the blurring process only on the background area without erroneous determination in the area determination of the subject and the background.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  In addition, the present invention is not limited to devices such as digital cameras, but includes built-in or external connection of imaging devices such as mobile phones, personal computers (laptop type, desktop type, tablet type, etc.), game machines, etc. It can be applied to any device. Therefore, the “imaging device” in this specification is intended to include any electronic device having an imaging function.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Focus lens 103 Optical system control part 104 Focus lens drive control part 105 Image pick-up element 106 Imaging system control part 107 Analog signal processing part 108 A / D conversion part 109 Digital signal processing part 110 Internal memory | storage part 111 I / F
112 Image display unit 113 System control unit 114 SW1
115 SW2
116 Main SW

Claims (8)

Acquisition means for acquiring a plurality of photographed images having different focal positions;
Area discriminating means for discriminating a subject area and a background area by comparing signals of the plurality of photographed images for a plurality of areas of the photographed image;
Blur processing means for performing blur processing on the captured image;
The photographed image subjected to the blur processing by the blur processing means and the photographed image not subjected to the blur processing are synthesized according to the area discrimination result obtained by the area discriminating means, and the background area is blurred. Combining means for generating an image,
The area discriminating unit, after performing the area discrimination with the size of the first area, for an area where the area discrimination result with the size of the first area is neither the subject area nor the background area, An image processing apparatus, wherein the area determination is performed with a size of a second area smaller than a size of the first area.   The image processing apparatus according to claim 1, wherein the area determination unit reflects an area determination result of a surrounding area for an area having a determination result different from the surrounding area.   The image processing apparatus according to claim 1, wherein the region determination unit performs the region determination using edge information of each region of the plurality of photographed images.   The image processing apparatus according to claim 3, wherein the region determination unit performs the region determination using color information of each region of the plurality of photographed images.   The image processing apparatus according to claim 3, wherein the area determination unit performs the area determination using face detection information detected from the captured image. An acquisition step of acquiring a plurality of captured images having different focal positions;
An area determination step for performing area determination for determining a subject area and a background area by comparing signals of the plurality of captured images for a plurality of areas of the captured image;
A blur processing step for performing blur processing on the captured image;
The photographed image that has been subjected to the blurring process and the photographed image that has not been subjected to the blurring process are synthesized according to the area discrimination result obtained in the area discrimination step to generate a blurred composite image of the background area And a synthesis step to
In the region determination step, after performing the region determination with the size of the first region, the region determination result with the size of the first region is not any of the subject region and the background region. An image processing method, wherein the area determination is performed with a size of a second area smaller than a size of the first area.   A computer-executable program in which the steps of the image processing method according to claim 6 are described.   A computer-readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 6.

Images