JP5434361B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a digital still camera.

  The area of an image sensor used in a compact digital camera is smaller than the area of an image sensor used in a single-lens reflex camera or a silver salt film. For this reason, the focal length of the photographing optical system necessary for photographing an image with the same angle of view is shorter than that of a single-lens reflex camera in a compact digital camera. If the focal length is short, the depth of field will be deep even if the F number of the photographing optical system is the same. Here, if the F-number can be reduced (brightened) in proportion to the shortening of the focal length, the depth of field can be reduced, but it is necessary to increase the diameter of the imaging optical system, and the size And the cost increases. For this reason, when a picture is taken with a compact digital camera, a relatively wide distance range is in focus. This is an advantage in that an image with less blur can be obtained when images of the same brightness are taken, but it is difficult to shoot with a large background blur as in portraits.

  For the above-described problems, Patent Documents 1 to 4 propose techniques for blurring the background by image processing. In these cameras, the main subject is extracted according to the distance and the position within the angle of view, and the background portion is blurred. In particular, in the techniques according to Patent Documents 1 and 2, the distance to the subject in each area of the screen is measured, the main subject is separated according to the distance, and the blurring process is performed with the blur strength according to the distance. A more natural image can be obtained.

  However, in the technique according to Patent Document 1, the main subject is determined based on the distance measurement result obtained by dividing the photographing surface in the vertical direction or the horizontal direction, and the non-main subject portion is blurred. Because it has a complicated shape, if the subject area is determined using only the distance measurement result, the problem is that the blur shape does not match the subject, i.e. A problem of missing blur in the main subject portion occurs. Further, in the blurring process of the technique according to Patent Document 1, since the blurring process is performed by continuously changing the number of data used for averaging according to the distance from the main subject, the process takes time. In addition, there is a problem that it is necessary to configure dedicated hardware and the cost increases.

  Furthermore, in the technique according to Patent Document 2, the distance to a plurality of points in the shooting target range is detected by the multi-focus method, and blurring processing is performed based on the detected distance. Therefore, depending on the type of the non-main subject, a correct distance measurement result cannot be obtained, and the non-main subject portion is blurred. Further, in the blurring process of the technique according to Patent Document 2, the blurring process is performed by changing the filter characteristics for each pixel so that the range corresponding to the difference in the circle of confusion obtained from the distance is dispersed. The problem that it takes a long time, and the problem that the cost increases because it is necessary to configure dedicated hardware.

  On the other hand, in Patent Document 5, distance measurement is performed by CCDAF and a main subject is extracted, so that a background (corresponding to a non-main subject) image is not required without requiring special hardware or processing. A technique for performing a blurring process is disclosed. According to such a technique, the blurring process can be performed with a simple configuration. However, in the technique according to Patent Document 5, although the contour extraction is performed using the distance measurement result in the determination of the main subject region, the background portion is not considered and the blur shape may not match the subject. is there. Therefore, higher accuracy in subject extraction is desired.

  The present invention has been made in view of the above situation, and provides an image processing apparatus, an image processing method, and a digital still camera that can obtain a high-quality blurred image regardless of a subject and can perform high-speed processing. It is intended to provide.

In order to solve the above problems, an image processing apparatus, an image processing method, and a digital still camera according to the present invention are specifically described in the following (1) to (7), (9) to (15), and (17). It has the following technical features.
(1): Image data including main subject image data and non-main subject image data, and distance information including a distance to the main subject and a distance to the non-main subject are input, and blur processing is performed on the image data. An image processing apparatus, a distance dividing unit that divides the distance information into divided distance information having a predetermined number of steps, and area determination of the image data based on luminance and / or chromaticity, and outputting an area determination result An area determination unit, a labeling unit that outputs the label data by associating the division distance information with the region determination result, and smoothing the image data by smoothing the image data with different processing according to the predetermined number of stages. Smoothing means for outputting; and a combining process for outputting combined image data obtained by combining the image data and the smoothed image data based on the label data. Comprising means, wherein the labeling means is in the small region of overlap in the boundary region including the main object image data, the image processing apparatus characterized by performing weighting to 0 divided distance information of the small area is there.

(2): The distance dividing unit divides the distance information into divided distance information composed of N stages,
The image according to (1), wherein the smoothing means smoothes the image data by (N-1) different processes and outputs (N-1) different smoothed image data. It is a processing device. (However, N is an integer of 2 or more.)

(3): The apparatus according to (1) or (2), further including a reduction unit that reduces the image data, wherein the region determination unit determines a region in the image data reduced by the reduction unit. An image processing apparatus.

(4) The image processing apparatus according to (3), further including an enlargement unit that enlarges the label data.

(5) The image processing apparatus according to any one of (1) to (4), further including a composite image data smoothing unit that smoothes the composite image data.

(6): The image processing apparatus according to any one of (1) to (4), wherein the synthesis processing unit synthesizes the image data and the smoothed image data by weighted addition. It is.

(7): comprising label data smoothing means for outputting smoothed label data obtained by smoothing the label data;
The image processing apparatus according to (6), wherein the combining processing unit combines the image data and the smoothed image data by weighted addition using the smoothed label data.

(9): An image processing method for performing blur processing on image data including input main subject image data and non-main subject image data, wherein the distance to the input main subject and the distance to the non-main subject are A distance dividing step of dividing the distance information consisting of a predetermined number of steps into a distance dividing step, a region determining step of determining the region of the image data based on luminance and / or chromaticity, and outputting a region determination result, A labeling step of associating the division distance information with the region determination result and outputting label data; and a smoothing step of smoothing the image data by different processing according to the predetermined number of steps and outputting smoothed image data And a synthesis processing step of outputting synthesized image data obtained by synthesizing the image data and the smoothed image data based on the label data. Labeling step, in a small region of overlap in the boundary region including the main object image data, an image processing method characterized by performing weighting to 0 divided distance information of the small area.

(10): The distance dividing step divides the distance information into N pieces of divided distance information, and the smoothing step smoothes the image data by (N−1) different processes, and (N The image processing method according to (9), wherein the smoothed image data different in (-1) is output. (However, N is an integer of 2 or more.)

(11): The method according to (9) or (10), further including a reduction process for reducing the image data, wherein the area determination process determines an area in the image data reduced in the reduction process. This is an image processing method.

(12) The image processing method according to (11), further including an enlargement step of enlarging the label data.

(13) The image processing method according to any one of (9) to (12), further including a composite image data smoothing step for smoothing the composite image data.

(14) The image processing method according to any one of (9) to (12), wherein the synthesis processing step combines the image data and the smoothed image data by weighted addition. It is.

(15): a label data smoothing step for outputting smoothed label data obtained by smoothing the label data, wherein the synthesis processing step performs weighted addition of the image data and the smoothed image data by the smoothed label data The image processing method according to (14), wherein the image processing method is combined.

(17): A digital still camera comprising the image processing apparatus according to any one of (1) to (7) above.

  According to the configuration described in the above (1), (2), (9) or (10), the main subject and the non-main subject (for example, the background) can be appropriately separated, and a high-quality blurred image can be obtained. As a result, blurring can be performed at high speed.

  According to the configuration described in the above (3), (4), (11) or (12), the blurring process can be further performed at higher speed.

  According to the configuration described in the above (5) or (13), a higher-quality blurred image can be obtained.

  According to the configuration described in (6) or (14) above, the joint between the original image data and the smoothed smoothed image data or between the smoothed image data smoothed with different intensities is made inconspicuous. And a high-quality blurred image can be obtained.

  According to the configuration described in (7) or (15) above, the joints between the original image data and the smoothed smoothed image data or between the smoothed image data smoothed with different intensities are more inconspicuous. And a higher-quality blurred image can be obtained.

  According to the configuration described in (17) above, it is possible to realize a digital still camera that performs high-quality and high-speed blurring.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a digital still camera that can obtain a high-quality blurred image regardless of a subject and can perform high-speed processing.

It is a block diagram which shows the structure of the digital still camera apparatus and connection apparatus which concern on this invention. It is a top view which shows the external appearance of one Embodiment in the digital still camera concerning this invention. It is a front view which shows the external appearance of the digital still camera of FIG. It is a rear view which shows the external appearance of the digital still camera of FIG. It is an operation | movement flow of 1st Embodiment and 2nd Embodiment in the digital still camera concerning this invention. It is a figure which shows the image processing of 1st Embodiment in the digital still camera concerning this invention. It is a figure which shows the change of AF evaluation value of 1st Embodiment in the digital still camera concerning this invention. It is a figure which shows the pixel used as calculation object by HPF of AF evaluation value of 1st Embodiment in the digital still camera concerning this invention. It is a figure which shows the distance to the to-be-photographed object and the predetermined range of AF area of 1st Embodiment in the digital still camera concerning this invention. It is explanatory drawing for demonstrating the area allocation of labeling of 1st Embodiment in the digital still camera concerning this invention. It is explanatory drawing for demonstrating the smoothing process and synthetic | combination process of 1st Embodiment in the digital still camera concerning this invention. It is a figure which shows distribution of the filter coefficient of 1st Embodiment in the digital still camera concerning this invention. It is explanatory drawing for demonstrating the bit allocation of the label data of 2nd Embodiment in the digital still camera concerning this invention. It is explanatory drawing for demonstrating the smoothing process and synthetic | combination process of 2nd Embodiment in the digital still camera concerning this invention. It is a sample figure of the image data input into the image processing device concerning the present invention. FIG. 16 is a diagram in which ranging results are superimposed on the sample diagram shown in FIG. 15. It is a block diagram which shows the structure in other embodiment of the image processing apparatus concerning this invention. It is a figure for showing the example of the division | segmentation of the area | region in the sample figure shown in FIG. It is a figure which shows the example of the processing flow of the area | region division in the other Embodiment 1 of the image processing apparatus concerning this invention. It is a figure which shows the detailed internal structure of the distance information acquisition part. FIG. 16 is a diagram in which face detection results are superimposed on the sample diagram shown in FIG. 15. It is a graph which shows the difference of the block distance of the erroneous ranging block E, and the block distance of a surrounding block. It is explanatory drawing which shows the arrangement | positioning relationship between the erroneous ranging block E and a surrounding block. It is a figure which shows an example of a 5x5 Gaussian filter. It is a figure which shows an example of the blur function DB110. It is a block diagram which shows the hardware constitutions of a digital camera. It is a figure which shows the example of the processing flow of the blurring process in other Embodiment 1 of the image processing apparatus concerning this invention. FIG. 10 is a block diagram illustrating a hardware configuration of an image processing apparatus according to another embodiment 2. It is a figure which shows the example of the process flow of the blurring process in other Embodiment 2 of the image processing apparatus concerning this invention.

  The image processing apparatus according to the present invention receives image data including main subject image data and non-main subject image data, and distance information including a distance to the main subject and a distance to the non-main subject, and the image An image processing apparatus that performs blurring processing on data, a distance dividing unit that divides the distance information into divided distance information having a predetermined number of steps, and area determination of the image data based on luminance and / or chromaticity Area determination means for outputting an area determination result, labeling means for outputting label data in association with the division distance information and the area determination result, and smoothing the image data with different processing according to the predetermined number of steps Smoothing means for outputting smoothed image data, and a composite image obtained by combining the image data and the smoothed image data based on the label data Characterized by comprising a synthesis processing means for outputting over data, the.

Next, the image processing apparatus, the image processing method, and the digital still camera according to the present invention will be described in more detail.
Although the embodiments described below are preferred embodiments of the present invention, various technically preferable limitations are attached thereto, but the scope of the present invention is intended to limit the present invention in the following description. Unless otherwise described, the present invention is not limited to these embodiments.

[First Embodiment]
Hereinafter, the first embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 shows the configuration of a digital still camera device and connection equipment according to the present invention. In FIG. 1, 01 is a digital still camera device. Reference numeral 02 denotes a system control unit including a CPU, a NAND flash memory, an SDRAM, a timer, and the like, which are provided to control the entire digital still camera device 01. Reference numeral 03 denotes an imaging unit that is provided for imaging and includes a motor for driving optical system components, a CCD drive circuit for driving a CCD, an A / D converter, and the like.

  04 performs various image processing on the image signal (image data) obtained by the imaging unit and controls the CCD drive timing and lens drive motor of the imaging unit 03 to perform zooming, focusing, exposure adjustment, etc. The image processing unit includes an image processing DSP (digital signal processor), a RAM, and the like, which are provided to compress and decompress images. 05 performs signal processing for displaying the image signal processed by the image processing unit 04 on the LCD, and generates various graphic images for the user interface and displays them on the LCD. A display control unit including an A converter, an on-screen display controller, and the like.

  Reference numeral 06 denotes an LCD provided for displaying an image and displaying a graphic for a user interface. Reference numeral 07 denotes a recording medium interface unit including a memory card controller or the like provided for an interface with the recording medium. Reference numeral 08 denotes a recording medium that is detachable from the digital still camera device 01 and includes a flash memory or the like provided to store a compressed image signal and various information associated with the image.

  Reference numeral 09 denotes a hard key interface unit composed of a sub CPU and the like, which is provided for detecting the state of a user interface such as a dial and controlling main power to the main CPU. Reference numeral 10 denotes a communication interface unit including a communication controller, which is provided for performing data communication through USB connection. Reference numeral 11 denotes a PC (personal computer) that is connected to the digital still camera device 01 via USB, transfers images from the digital still camera device 01, plays back, and makes various settings to the digital still camera device 01.

  Reference numeral 12 denotes a mode dial for setting a photographing mode. Reference numeral 13 denotes a release key for instructing photographing. Reference numeral 14 denotes a CCD for imaging. Reference numeral 15 denotes a lens for forming an optical image on the CCD 14.

FIG. 2 is a top view showing an appearance of an embodiment of the digital still camera of the present invention. FIG. 3 is a front view showing an appearance of the digital still camera of FIG. FIG. 4 is a rear view showing the appearance of the digital still camera of FIG.
2 to 4, a release shutter (SW 1) 62, a mode dial (SW 2) 63, and a sub LCD 64 are arranged on the upper surface of the digital still camera body 1. An SD card / battery cover 65, a strobe light emitting unit 66, an optical viewfinder 67, a remote control light receiving unit 69, and a lens barrel unit 70 are arranged on the front surface of the digital still camera body 1. On the back of the digital still camera body 1, there are an AF LED 71, a strobe LED 72, a zoom switch (wide) (SW3) 73, a zoom switch (remote) (SW4) 74, a self-timer / delete switch (SW5) 75, a menu switch (SW6). 76, an up / strobe switch (SW7) 77 is arranged. Further, on the back of the digital still camera body 1, a right switch (SW8) 78, a display switch (SW9) 79, a down / strobe switch (SW10) 80, a left / image confirmation switch (SW11) 81, an okay switch (SW12) 82, an LCD monitor 83, and a power switch (SW13) 84 are arranged. Furthermore, an optical viewfinder 67 is disposed on the back surface of the digital still camera body 1 at the same position as the front surface. The optical finder 67 is fixed in a state where it is inserted from the front side to the back side of the apparatus.

  First, a conventional startup operation will be described. When the user presses the power button 84, the hard key interface unit 09 turns on the power supply to the main CPU. The main CPU in the system control unit 02 first starts access (program execution) from the boot unit of the NAND flash memory, and transfers program data to the SDRAM by the boot program. When the transfer to the SDRAM is completed, the program execution pointer (program counter) is moved to the transferred program on the SDRAM, and thereafter, the startup process is started by the program on the SDRAM.

  The startup processing includes OS (operating system) initialization, lens barrel extension processing, recording media initialization processing, and the like. The lens barrel feeding process is performed by applying a pulse signal to the lens driving motor of the imaging unit 03 at predetermined intervals (2 mS) via the image processing unit 04. Also, in the initialization process of the recording medium 08, after supplying power and a clock to the recording medium 08 via the recording medium interface unit 07, an initialization command is issued to the recording medium 08. The actual initialization process is performed in the recording medium 08, and the system control unit 02 polls the status of the recording medium 08 at intervals of 10 mS in order to detect the completion.

  Next, the operation during shooting will be described. Prior to shooting, the user operates various keys and dials shown in FIGS. 2 to 4 to determine a shooting mode (high image quality mode, low image quality mode, etc.). The user's operation content is determined by the system control unit 02 through the hard key interface unit 09, and the system control unit 02 generates a guidance graphic to the display control unit 05 according to the operation, and prompts the user to perform the next operation. When the photographing mode is determined, the system control unit 02 sets processing parameters corresponding to the mode in the image processing unit 04.

  Alternatively, the user operates the zoom switches 73 and 74 to determine the angle of view (composition). The user's operation content is determined by the system control unit 02 through the hard key interface unit 09, and the system control unit 02 controls the imaging unit 03 in accordance with the operation to drive the lens. In accordance with control from the image processing unit 04, the imaging unit 03 starts an imaging operation for displaying a monitoring image prior to actual shooting.

  The imaged data is continuously sent to the image processing unit 04. The image processing unit 04 performs processing such as color space conversion, gamma correction, and white balance adjustment, and then sends the image data to the display control unit 05. In the display control unit 05, the image data is signal-processed and displayed on the LCD 06, and the imaging state is presented to the user. When the release button 13 is pressed, the operation is discriminated by the system control unit 02 through the hard key interface unit 09 as in the mode setting.

  The imaging unit 03 performs focus adjustment according to the control from the image processing unit 04 and then sends the captured image to the image processing unit 04. The image processing unit 04 performs image processing and compression processing according to the shooting mode. The system control unit 02 reads the compressed image data, adds header information, and then writes the compressed image data to the recording medium 08 through the recording medium interface unit 07. This completes a series of shooting operations.

Next, the operation flow of image processing that is a feature of the present invention will be described in more detail.
First, according to the first embodiment of the present invention, the original image data is smoothed to create one surface of the smoothed image data, and is synthesized with the original image data. This is repeated as necessary (this embodiment). An example will be described in which this is repeated three times.

The operation flow is shown in FIG. This operation flow shows an operation at the time of shooting after the release button 13 is pressed.
When the photographing operation is started, the system control unit 02 controls the image processing unit 04 and the imaging unit 03 to perform a CCDAF scanning operation (step 01-001).
Subsequently, the system control unit 02 determines the peak position for each position in the imaging region (step 01-002).

Here, an outline of CCDAF will be described.
In general, in an electronic image pickup apparatus having a two-dimensional image pickup device such as a digital camera or a video camera, a sharpness of a screen is detected by a video signal photoelectrically converted by the image pickup device, and a focusing lens is set so that the sharpness is maximized. The focus is adjusted by controlling the position of. Normally, the sharpness is small when the focus is blurred, increases as the focus is achieved, and reaches a maximum value when the focus is completely achieved.
The CCDAF gradually moves the focus lens from the infinite end to the near end, images the subject at a plurality of positions, and the position of the focus lens from which the image with the highest sharpness is obtained among the plurality of imaged images. Is a method of setting the in-focus position.

Hereinafter, a description will be given with reference to FIG.
In FIG. 6A, reference numeral 100 denotes an entire image capturing area, and reference numeral 101 denotes one AF evaluation value area.
As shown in FIG. 6A, the AF evaluation value area is a small area of 16 × 12 obtained by equally dividing the photographing area, and the AF evaluation value (small area 101) for each small area 101 is determined by CCDAF. (The integrated value of the contrast of the image).

The system control unit 02 analyzes the AF evaluation value for each lens position obtained by the CCDAF scan for each small region 101 based on a predetermined algorithm, and determines the lens position corresponding to the peak position of the AF evaluation value. judge.
Further, the system control unit 02 converts the driving position of the lens from the current zoom position into distance information for each small area 101.

An example of the relationship between the lens position (focus distance) and the AF evaluation value in this CCDAF is shown in FIG.
In FIG. 7, the horizontal axis indicates the lens position and the focusing distance corresponding to the lens position, and the vertical axis indicates the AF evaluation value.
Reference numeral 901 denotes a waveform indicating a change in AF evaluation value associated with a scan of a subject at a long distance (for example, the sky portion 102 in the background in FIG. 6C, which is a non-main subject), and 902 is a subject at the same intermediate distance. For example, a waveform indicating a change in AF evaluation value accompanying a scan of the background curb portion 103 in FIG. 6C, which is a non-main subject, and reference numeral 903 denotes an object at a short distance (for example, FIG. It is a waveform showing the change of the AF evaluation value accompanying the scanning of the human part 104) in C).
Here, the AF evaluation value is obtained by performing HPF (high pass filter) calculation between the pixels in the horizontal direction for each pixel in the AF evaluation value area 101 and adding the obtained high frequency components.

As the HPF coefficient, for example, a value such as ki = {− 1, −2, 6, −2, −1} is used.
k0 is a coefficient to be multiplied by the pixel of the pixel in the horizontal direction-2 of the target pixel, k1 is a coefficient to be multiplied to the pixel of the pixel of interest in the horizontal direction-1 coordinate, k2 is a coefficient to be multiplied by the target pixel, and k3 is A coefficient to be multiplied by a pixel having a coordinate in the horizontal direction + 1 of the target pixel, and k4 is a coefficient to be multiplied by a pixel having a coordinate in the horizontal direction + 2 of the target pixel.

The calculation of the AF evaluation value will be specifically described with an example in which five pixels in the AF evaluation value area shown in FIG. 8 are extracted.
In FIG. 8, reference numeral 1001 denotes a pixel having a coordinate in the horizontal direction-2 of the target pixel, 1002 a pixel having a coordinate in the horizontal direction-1 of the target pixel, 1003 a target pixel, and 1004 a horizontal pixel of the target pixel + 1. A coordinate pixel 1005 indicates a pixel having a coordinate in the horizontal direction +2 of the target pixel. The AF evaluation value is obtained by the following formula (1).

  (AF evaluation value) = k0 × (value of pixel 1001) + k1 × (value of pixel 1002) + k2 × (value of pixel 1003) + k3 × (value of pixel 1004) + k4 × (1005) Pixel value) (1)

In order to obtain distance information from the AF evaluation value, a Gaussian imaging equation represented by the following equation (2-1):
1 / a + 1 / b = 1 / f Expression (2-1)
From the above, it is obtained as the following formula (2-2).
a = bf / (b−f) Expression (2-2)

  Here, a is the distance from the lens to the subject, b is the distance between the lens and the image sensor, and f is the focal length of the lens. The distance information from the lens to the subject is the distance information to be obtained. The focal length f of the lens is uniquely obtained from the zoom position at the time of shooting. The distance b between the lens and the image sensor is uniquely determined from the position of the focus lens where the peak of the AF evaluation value is obtained. As described above, it is possible to obtain distance information for each AF evaluation value area 101 of the entire region within the angle of view 100.

  For areas where the contrast of the subject is low and no peak is obtained, it is preferable to perform the complementing process at the peak position of the surrounding area.

  As described above, since the actual distance information is uniquely obtained from the peak position of the AF evaluation value, the following description will be made assuming that the peak position is synonymous with the distance.

The value in each area (AF evaluation value area 101) in FIG. 6A indicates the peak position in each area determined in step 01-002.
Subsequently, the system control unit 02 divides the distance measurement result (peak position; distance information) into a predetermined number of stages N (N = 4 in the present embodiment) as division distance information (step 01-003; distance) Division process). The predetermined number of stages N is an arbitrary integer of 2 or more. At this time, as N becomes larger, the blurred image becomes higher quality, but if N becomes too large, processing takes time, which is not preferable.

In step 01-002, the system control unit 02 first determines the AF evaluation value area 101 within the predetermined range and the distance to the subject (main subject) in the AF area 107 as a main subject block.
In FIG. 6 (A), 107 indicated by shading indicates an AF area. The AF area 107 does not need to be the center in the angle of view 100, and the user can specify an arbitrary position in the angle of view 100 corresponding to the main subject to be imaged.

The distance to the subject in the AF area 107 and the predetermined range for determining the main subject block will be described with reference to FIG.
In FIG. 9, the horizontal axis indicates the distance from infinity to the short distance, 1101 to 1105, and 1108 indicate the distance of each part obtained by measuring the subject as illustrated in FIG. 6C in step 01-002. .

  1101 is the distance to the sky part 102 of the background, 1102 is the distance to the curb part 103 of the background, 1103 is the distance to the head of the person 104, 1104 is the distance to the face of the person 104, 1105 is , The distance to the chest of the person 104. Reference numeral 1108 denotes the distance between the background townscape 105 and the sea 106.

  As shown in FIG. 6A, the AF area 107 is set on the face of a person, and the distance to the subject in the AF area 107 is equal to 1104.

Reference numerals 1106 and 1107 denote predetermined ranges for determining the main subject block. Reference numeral 1106 denotes a range on the near side, and 1107 denotes a range on the far side.
In the example of FIG. 9, 1103, 1104, and 1105 are included in the predetermined range.
Based on the above determination, in this embodiment, the area indicated by 114 in FIG. 6B is determined as the main subject block.

The predetermined ranges 1106 and 1107 are set by referring to a table stored in advance in the system control unit 02 based on the focal length of the lens and the distance to the subject in the AF area.
When the focal length of the lens is long, the predetermined range is short, when the focal length of the lens is short, the predetermined range is set long, and when the distance to the subject in the AF area is long, the predetermined range is When the distance to the subject in the AF area is short and the predetermined range is short, the predetermined range is set.
Alternatively, as another method, it may be automatically determined from the distribution of the distance measurement result (distance information) value in FIG.

A block in which the value of each area (AF evaluation value area 101) in FIG. 6B is 0 is a main subject block.
For the background portion which is a non-main subject in the present embodiment, the value is divided into the remaining three stages from the distribution of the distance measurement result (distance information) values in FIG. 6A in the area other than the main subject block. .
In the present embodiment, the region where the value is less than 11 (the value in FIG. 6B is 3), the region where the value is 11 or more and less than 15 (the value in FIG. 6B is 2), and the value is 15 or more. Is divided into three stages (the value in FIG. 6B is 1).
In this embodiment, the non-main subject is composed only of the background. However, the non-main subject in the present invention may be composed only of the foreground of the main subject, and is composed of the background and the foreground. Also good.

  Subsequently, the system control unit 02 controls the image processing unit 04 to reduce the captured image (image data) as a separate image to a predetermined image size (320 × 240 in the present embodiment) (step 01-004; reduction process) Then, region determination is performed based on the image obtained by reduction (step 01-005; region determination step).

  The region is determined based on the brightness and color difference of the image data, and when the difference between adjacent pixels is less than a predetermined value (10 in the present embodiment), the region is determined as the same region (formula (3) below) reference.). In this embodiment, the area is determined based on the luminance and color difference of the image data. However, the area determination in the present invention may be performed based on the luminance or color difference of the image data.

abs (Y1-Y0) + abs (U1-U0) + abs (V1-V0) <10
... Formula (3)

In the above equation (3), Y0, U0, and V0 are the luminance or color difference of the reference pixel, and Y1, U1, and V1 are the luminance or color difference values of adjacent pixels.
abs () indicates processing for obtaining the absolute value of the value in parentheses.
By performing the determination while moving the position of the reference pixel to the entire image, it is possible to determine an area close to brightness and color.

FIG. 6D shows a region determination result that is a result of the region determination step in step 01-005. In FIG. 6D, each area surrounded by a thick line and / or a thin line is a result of area determination in which image data is grouped based on luminance and color difference.
In this embodiment, as will be described later, blur processing is performed on all the background areas that are non-main subjects, but blur processing is performed on non-main subjects whose positions from the main subject are within a predetermined range. It is desirable not to do so. In other words, it is desirable to perform the blur process on the non-main subject region whose position from the main subject is outside the range of the predetermined distance in the non-main subject region.

Subsequently, the system control unit 02 performs area allocation (labeling) (step 01-006; labeling step).
The labeling is determined by superimposing each area determined in step 01-005 (FIG. 6D; area determination result) and the distance measurement result divided in step 01-003 (FIG. 6B; divided distance information). To do. That is, each pixel extracted in the region determination step counts which of the N-stage division distance information overlaps with each other, and labeling data in which each area is assigned to the division distance information of the stage with many overlaps. Is output.

This labeling method will be specifically described with reference to FIG.
In FIG. 10, 700 surrounded by a thick line is an AF evaluation value area with division distance information 0 (first stage), 701 is an AF evaluation value area with division distance information 1 (second stage), and 702 is a division. An AF evaluation value area with distance information 2 (third stage), and 703 similarly indicate an AF evaluation value area with division distance information 3 (fourth stage). These regions 700, 701, 702, and 703 are regions surrounded by thick lines as shown in the figure, and are composed of pixels 705 surrounded by 20 × 20 thin lines. As described above, since the area determination is performed with an image reduced to 320 × 240, when overlapped with the 16 × 12 AF evaluation value area, each AF evaluation value area includes 20 × 20 pixels.
A circular area 704 surrounded by a wavy line is one area indicated by the overlapped area determination result.

  The system control unit 02 counts the number of pixels included in each of the AF evaluation value areas 700 to 703 overlapping in the area 704, and assigns the area 704 to the division distance information having the largest number of counted pixels. At this time, a predetermined weighting (in the present embodiment, counting by 3) is performed on a pixel that overlaps the AF evaluation value area (main subject area) whose division distance information is 0, and the determination is made. In the present embodiment, when there are a plurality of pieces of division distance information having the largest number of pixels counted in each area, assignment close to 0 is preferentially performed.

A specific calculation of the present embodiment in the example of FIG. 10 will be described.
The number of pixels in the area where the division distance information is 0 is 77 pixels, which is equivalent to 231 pixels because the weight is tripled. The number of pixels in the area where the division distance information is 1 is 120 pixels, and the number of pixels in the area where the division distance information is 2 The number of pixels in the area of 204 pixels and the division distance information 3 is 143 pixels, and the distance measurement result 0 is assigned to the area 704. In the allocation, label data equal to the number of pixels (320 × 240) used for area determination is created, and the value of label data corresponding to the position of the area 704 is set to zero.

By weighting the area whose division distance information is 0, a small area is generated in the area determination result, and the small area is set to 0 when it overlaps the boundary of the division distance information. As a result (when the small area is the main subject), it is possible to prevent the main subject from being blurred.
In this embodiment, the main subject is weighted three times, but other values may be used. Alternatively, in the area to be allocated, when there is an overlap of “1” or more with “0” that is the division distance information corresponding to the main subject area, a method of setting the label value “0” in the area is also effective. is there.

  Similarly, in the determination, when a region with division distance information 1 is assigned, label value 1 is assigned. When a region with division distance information 2 is assigned, label value 2 is given, and region with division distance information 3 is given. Is assigned, label value 3 is set to generate 320 × 240 label data.

  In the example of FIG. 6, the area surrounded by the thick line in FIG. 6D shows the label data setting result, and the area 108 (corresponding to the main subject area in this embodiment) is 0. However, 1 is set in the areas 109 and 110, 2 is set in the areas 111 and 112, and 3 is set in the area 113.

  As described above, the processing in steps 01-005 and 006 is performed based on the reduced image. Since complicated processing as described in steps 01-005 and 006 needs to be performed by the CPU, it can be performed at high speed by performing it based on the reduced image.

  Subsequently, the system control unit 02 enlarges the label data (step 01-007; enlargement process). Enlargement is performed so that intermediate values other than 0, 1, 2, and 3 are not generated by the nearest neighbor method, and the image is enlarged to the same size as the size of the captured image data.

Hereinafter, the smoothing step and the synthesis processing step will be described with reference to FIG.
The system control unit 02 controls the image processing unit 04 to generate a copy of the captured image data (original image data) (step 01-008, FIG. 11 (1)), and further, the captured image data (original image data) A blurred image (smoothed image data) is generated (step 01-009, FIG. 11 (2); part of the smoothing step).

Here, the details of the blurring process (generation of smoothed image data) will be described based on the flow of FIG.
Based on the label value of the label data, the system control device 02 determines the processing contents as shown in Table 1 below (step 03-001).

In Table 1, the label value indicates the value of the label data (a larger value causes greater blur), and the reduction process indicates the reduction ratio (the ratio of the length of one side of the image) of the reduction process determined by the label value. Yes. That is, when obtaining a large blur, the spatial filter process is performed after the image is further reduced. In the reduction processing at this time, sampling is performed by the bilinear method so as to reduce the number of pixels of the image data.
Note that the denominator value 16 of the reduction ratio of the reduction process is a value that is a common divisor of the horizontal size and vertical size of the image, and the reduced image size has a fractional part in both the horizontal and vertical directions. Does not occur.

  As a result, errors due to rounding in the reduction processing and enlargement processing can be eliminated, the image size after reduction and enlargement can be accurately adjusted to the original image, and as a result, the image quality of the processing result of the synthesis processing described later is improved. can do.

When the label value is 1, only the spatial filter process (blurring process) is performed without performing the reduction process (steps 03-002, 003).
Spatial filter processing is executed by the image processing unit 04 based on settings from the system control unit 02.
In the spatial filter processing, the filter coefficient (k (ix, iy)) is calculated for the input image (In (x, y)) as shown in the following equation (4) to obtain the output image (Out (x, y)). .

Here, In: input image, Out: output image, k: filter coefficient, fs: filter size (7 in this embodiment). In Equation 4, the coordinate calculation results (x + ix−fs / 2 and y + iy−fs / 2) are clipped so as to be rounded down to an integer and point to the input image.
Next, an example of the filter coefficient is shown in FIG.

When the label value is 2 or more, after the image is once reduced, the spatial filter process (blurring process) is performed, and then the enlargement process is performed to restore the original size (steps 03-004 to 006).
In the enlargement process, the enlargement process corresponding to the reciprocal of the reduction ratio performed in step 03-004 is performed, and as a result, the size of the image data becomes the original size.
In the enlargement process, sampling is performed so as to increase the number of pixels of the image by the bilinear method.

  These reduction, spatial filter, and enlargement processes are executed by the image processing unit 04 based on settings from the system control unit 02. In addition, since the area where the label value is 0 is an area corresponding to the main subject, there is no corresponding blurring process, and there is a corresponding blurring process only when the label values are 1, 2, and 3. That is, when divided into N stages, N-1 different smoothing processes are performed to generate N-1 smoothed image data.

Hereinafter, the description will return to the flow of FIG.
In the synthesis process (step 01-010; synthesis process step), the image (smoothed image data) subjected to the blurring process in step 01-009 is synthesized with the copy image (original image data) generated in step 01-008.
Compositing is performed based on the label value of the label data, and the copy image is overwritten with the pixel at the position where the label value of the image (one of the smoothed image data) subjected to the blurring process corresponding to the label value 1 is 1 Process (FIG. 11 (3)).
The same processing as described above is repeated for the label values 2 and 3 (steps 01-009 to 011 and FIGS. 11 (4) to (7)) to obtain composite image data.

Subsequently, the system control unit 02 performs a spatial filter process on the pixel located at the boundary between the label values (step 01-012, FIG. 11 (8); synthesized image data smoothing step).
This spatial filter processing (blurring processing) may be the same processing as step 03-003, or may be simplified processing in which all the filter coefficients in FIG.
By this processing, the joint between the regions having different label values can be made inconspicuous.
Finally, the image subjected to the above processing is compressed and recorded in the same manner as in the prior art, and the processing is completed (step 01-013).

[Second Embodiment]
Next, the second embodiment of the present invention, the original image data is smoothed to create smoothed image data (multiple planes as required, three planes in this embodiment), and then weighted addition An example of combining with original image data will be described.

The operation flow is shown in FIG. This operation flow shows the operation at the time of shooting after the release button 13 is pressed, as in the first embodiment. Note that description of other operations is omitted.
Further, since the operation of step 02-001 to 005 is the same as the operation of step 01-001 to 005, the description thereof is omitted.

  In step 02-006, the system control unit 02 performs area allocation (labeling), but the format of the label data is different from that of the first embodiment.

An example of the format of the label data is shown in FIG.
FIG. 13 shows the bit allocation of label data corresponding to one pixel, and the upper 2 bits (bit7, bit6) are the label value having the same contents as the first embodiment, and the lower 6 bits (bit5 to bit0). Indicates a coefficient of weighted addition described later.
Other operations (such as counting pixels overlapped in a region and assigning them to division distance information with a large number of pixels) are the same as those in the first embodiment, and thus the description thereof is omitted.

  As in the first embodiment, the system control unit 02 assigns a label value of 0 when a region with division distance information of 0 is assigned to each pixel of each region, and a division distance information of 1. When an area is assigned, label value 1 is assigned. When an area with division distance information 2 is assigned, label value 2 is assigned. When an area with division distance information 3 is assigned, label value 3 is assigned. Create the set label data. At the same time, the coefficient part of the weighted addition is initialized to zero.

Subsequently, the system control unit 02 performs spatial filter processing (smoothing) on the label data to generate smoothed label data (step 02-007; label data smoothing means).
This processing may be the same processing as step 03-003, or may be simplified processing in which all the filter coefficients in FIG.

By this processing, the upper 2 bits of the label value are distributed to the weighted addition unit, and when the label data is viewed as data with a decimal point between bits 6 and 5, intermediate values such as 1.5 and 2.5 are obtained. Will be generated.
Thereby, in the weighted addition mentioned later, the joint between each area | region where a label value differs can be made not conspicuous.

Further, the system control unit 02 enlarges the smoothed label data (step 02-008; enlargement step). Unlike the first embodiment, the enlargement is performed by the bilinear method.
As a result, the coefficient values of the weighted addition are further dispersed, and in the weighted addition described later, the joints between regions having different label values can be made less noticeable.

Hereinafter, the smoothing step and the synthesis processing step will be described with reference to FIG.
The system control unit 02 controls the image processing unit 04 to generate smoothed image data that is a blurred image of captured image data (original image data).
The smoothing process is the same as in the first embodiment, and is repeated for the label values 1, 2, and 3 (step 02-009 to 010), and each of the blurred images (N-1 different smoothing) is performed. Image data (three in the case of the present embodiment) is generated (FIGS. 14 (1) to (3)).

In the composition processing step (step 02-011), the photographed image data (original image data) and the three smoothed image data (FIGS. 14 (1) to (3)) generated in steps 02-009 to 010 are combined.
The synthesis is performed based on the label value of the label data and the weighted addition coefficient, that is, the smoothed label data.

  When the label value is 0, weighted addition is performed between the captured image data (original image data) and the image smoothed with the label value 1 (FIG. 14 (1)) by the following equation (5-1). Do.

Pixel value of composite image data = ((pixel value of original image data × (64−weighted addition coefficient)) + (pixel value of smoothed image data smoothed with label value 1 × weighted addition coefficient)) / 64
... Formula (5-1)

  When the label value is 1, the following expression is applied between the image smoothed with the label value 1 (FIG. 14 (1)) and the image smoothed with the label value 2 (FIG. 14 (2)). Weighted addition is performed according to (5-2).

Pixel value of composite image data = ((pixel value of smoothed image data smoothed with label value 1 × (64−weighted addition coefficient)) + (smoothed image smoothed with label value 2) Pixel value of data × weighted addition coefficient)) / 64
... Formula (5-2)

  When the label value is 2, the following expression is applied between the image smoothed with the label value 2 (FIG. 14 (2)) and the image smoothed with the label value 3 (FIG. 14 (3)). Weighted addition is performed according to (5-3).

Pixel value of composite image data = ((pixel value of smoothed image data smoothed with label value 2 × (64−weighted addition coefficient)) + (smoothed image smoothed with label value 3) Pixel value of data × weighted addition coefficient)) / 64
... Formula (5-3)

  When the label value is 3, (pixel value of composite image data = pixel value of smoothed image data smoothed with label value 3).

By the above synthesis processing step, the joint between the regions having different label values can be made inconspicuous.
Finally, the image subjected to the above processing is compressed and recorded in the same manner as in the prior art to complete the processing (step 02-012).

  As described above, the original image data is smoothed, one surface of the smoothed image data is created and synthesized with the original image data, and this is repeated as necessary (first embodiment), and the original image An example (second embodiment) in which data is smoothed and smoothed image data is created (multiple planes as required, three planes in this embodiment) and then synthesized with the original image data by weighted addition. Although two embodiments have been described, these differences will be further described.

  The first embodiment can use less memory than the second embodiment. In FIG. 11, in the first embodiment, (1), (3), (5), (7), (8) use the same memory area, and (2), (4), (6) can use the same memory area, whereas the second embodiment requires three-plane memories (FIGS. 14 (1) to (3)) of the same size as the original image data. If there are two memory areas, it can be processed.

  On the other hand, the second embodiment can obtain higher quality image quality than the first embodiment. While the first embodiment corrects the joint for the synthesized image by a spatial filter, the second embodiment generates label data in consideration of the joint in advance. The joints between the different regions can be processed more naturally.

  In each of the first and second embodiments described above, the example of determining the processing content (blurring amount) of the blurring process based on the label value has been described, but further, the distance measurement from which the label value is based The processing content (blurring amount) of the blurring process may be determined based on the result (FIG. 6A). In that case, it can be realized by holding a processing content determination table corresponding to Table 1 in the system control unit 02 corresponding to the distance range for determining each label value described in step 01-003.

  Further, the user may be able to set the blur intensity. In that case, it can be realized by holding a processing content determination table corresponding to Table 1 in the system control unit 02 according to the blur intensity set by the user.

Next, another embodiment (second invention) of the present invention will be described.
An image processing apparatus according to another embodiment of the present invention is an image processing apparatus that receives image data and distance information and performs a blurring process on the image data, and the image data is converted into luminance and / or color. Area dividing means for dividing the image data into a plurality of areas based on the degree, block dividing means for dividing the image data into a plurality of blocks, and a block distance setting for setting the block distance for each of the plurality of blocks based on the distance information Means, a main subject region extracting means for determining the plurality of regions based on the block distance and dividing the main subject region into a main subject region and a non-main subject region, and the main subject region and the non-subject in the plurality of blocks. For a block having a boundary line with the main subject area, the pixels in the main subject area in the block are kept at the distance of the pixel. Sets a first correction distance based on a block distance in a block belonging to the main subject region, and a pixel in the non-main subject region in the block is adjacent to the pixel distance or adjacent to the block. And a correction distance setting means for setting a second correction distance based on a block distance in a block belonging to the non-main subject area, and based on the block distance, the first correction distance, and the second correction distance. And a blur processing means for performing blur processing on the image data.
In addition, an erroneous ranging block determination unit that determines an erroneous ranging block that is a block belonging to the non-main subject area among the plurality of blocks and in which an appropriate block distance is not set due to erroneous ranging, and An erroneous ranging block correction unit that corrects the block distance of the erroneous ranging block based on the block distance of a block that is a peripheral block of the erroneous ranging block and belongs to the non-main subject area, and sets a corrected block distance. Preferably, the blur processing unit performs blur processing on the image data based on the block distance, the first correction distance, the second correction distance, and the correction block distance.
Hereinafter, other embodiments of the present invention will be described in more detail with reference to the drawings.

[Other Embodiment 1]
FIG. 15 is a sample diagram of image data input to the image processing apparatus, which is divided into 7 × 10 blocks. A represents object 1, B represents object 2, and C represents object 3, where A is the main subject and B and C are non-main subjects.
FIG. 16 is a diagram in which the distance measurement results are superimposed on the sample diagram shown in FIG. A contour block D in the contour of the person object 1 represented by A in FIG. 16 is a block in which A and B and / or C overlap. In other words, the contour block D is a block having a boundary line between a main subject region and a non-main subject region (described later).
Since a plurality of subjects overlap in the contour block D, the block distance (described later) of the contour block D is not accurate. For this reason, if the block distance is applied as it is to the contour block D, blurring and blurring occur. Further, the erroneous ranging block E belonging to the non-main subject cannot be accurately measured due to the influence of the subject's movement or the luminance value, and an appropriate block distance is not set.

FIG. 17 is a block diagram showing a configuration in another embodiment of the image processing apparatus according to the present invention.
An image processing apparatus according to the present embodiment includes an image input unit 101, an image region dividing unit 102, a main subject extracting unit 103, an image blur processing unit 104, an image composition processing unit 105, an image output unit 106, and a camera parameter input unit 107. A distance information acquisition unit 108, a blur function acquisition unit 109, a blur function DB 110, a main subject surrounding erroneous ranging block correction unit 111, and an erroneous ranging block correction unit 112 for an area other than the main subject.
Next, each part will be described.

(Image input unit 101)
In the image input unit 101, image data is input by an imaging device such as a digital camera or an image input device such as a scanner.

(Image area dividing unit 102)
An image area dividing unit 102 which is an area dividing unit divides image data into a plurality of areas by image processing.
By using information such as the edge and color of the image, pixels having similar image contrast and color characteristics are grouped into a single region, and the image data is divided into a plurality of regions. As shown in FIG. 18, it is divided into a block for collecting pixels having similar colors and contrast, A object 1, B object 2, and C background area.
As shown in FIG. 18, the image data area is divided by extracting areas of almost the same color as one lump. The difference in color between adjacent pixels in the image data is calculated. Alternatively, a method of extracting a block having substantially the same contrast as one region may be used.

FIG. 19 is a diagram showing an example of a process flow of area division in another embodiment of the image processing apparatus according to the present invention. Each process step04-1-5 is demonstrated in order.
Step04-1: Image color space conversion (YUV conversion)
First, the image is color space converted, and the RGB image is converted to YUV. Here, Y is the luminance of the image, and U and V represent the color characteristics of the pixel. U is the difference between the luminance signal and the B signal, and V is the difference between the luminance and the R signal.

step 04-2: Filter processing Next, filtering processing is performed on each of the Y, U, and V components. The filtering process averages noise components and small areas that change rapidly, making it easier to create a lump with the same characteristics. Any filtering filter may be used as long as noise removal and a smoothing effect can be obtained. For example, Gaussian having a smoothing effect may be used.

step04-3: region generation If the difference between the adjacent Y, U, and V components is smaller than a certain threshold, it is determined that they are the same lump, and a lump is generated. At this time, threshold values are set for Y, U, and V, respectively. The threshold value is adjusted and set in advance by experiment. Each threshold value is stored in the memory in advance.
In this way, a small area with a small characteristic difference is generated. At this time, the region may be divided by the threshold values of U and V that are color components, or the region may be divided by the respective threshold values including the threshold value of the luminance value Y.
As shown in FIG. 18, an object A, an object B, and a background C that are the same color block are separated.

step04-4: Deletion of micro area In some cases, an extremely small micro area may be generated due to the influence of noise or isolated points. Here, a threshold for the area of the area is provided, and a small area smaller than the threshold area is merged with the adjacent area. When merging, it is combined with the area with the least brightness and color difference with the adjacent area. The threshold for the minimum area can be adjusted and set in advance.

Step 04-5: Output region division result After the minute region is deleted, the fused region is output. The fused region is a fused region that is close in color or has little change in contrast.
As shown in FIG. 18, objects A, B, and background C, which are the same color and contrast mass, are separated.

(Distance information acquisition unit 108)
The distance information acquisition unit 108 divides the image data into a plurality of blocks, and acquires distance information (block distance) for each of the plurality of blocks. (Block division means, block distance setting means)
In this embodiment, the image data is equally divided into 7 × 10 squares, but it goes without saying that the number and shape of a plurality of blocks are arbitrary.

FIG. 20 shows a detailed internal configuration of the distance information acquisition unit 108.
A reduced image input unit 201 inputs image data at a plurality of different focal positions, and a lens focus adjustment unit 205 that adjusts the focus of the camera lens adjusts the focus of the lens.
The edge information calculation unit 202 divides the input image data of a plurality of different focal points into a plurality of image blocks, and obtains an average value of contrast values in each block. The contrast value of the block at the same position in a plurality of image data is compared, and it is determined that the image with the highest contrast value is focused on the block. From the focal position of the image frame, the distance information calculation unit 203 calculates the shooting distance (block distance). When shooting distance information for all blocks is calculated, distance information (block distance) for each block can be acquired. The distance information output unit 204 outputs the acquired distance information (block distance).
The distance information acquisition unit is not limited to this form, and the distance information may be acquired using another distance measurement sensor.

(Main subject extraction unit 103)
A main subject extraction unit 103 as a main subject extraction unit combines small areas divided by the image area division unit 102 and extracts meaningful objects. An object means the foreground, background, and person of an image. When extracting an object, color information, edge information, and distance information can be used. As shown in FIG. 15, when distance information can be obtained in units of blocks, an object is extracted by combining the divided small areas using the distance information as a reference value. For example, in the area division processing, the head and body of a person may be divided into two areas, each of which is a small area. In distance information, the distance between the block in the head area and the block in the body area is approximately the same. And body are extracted as one object. Also, because the distance between adjacent trees is different, it is extracted as another object.

A meaningful object means that the foreground and the background are distinguished. Since the present invention aims to blur non-main subjects, the foreground and background can be distinguished, and the distance between objects need only be known.
As shown in FIG. 18, if the distance difference between the object A, the object B, and the background C is known, they can be separated.
Since the object regions A, B, and C are created by the image area dividing unit 102, if the difference between the distances of the objects A, B, and C is known from the distance information, each of them may be blurred.
Therefore, as shown in FIG. 15 of the drawing, if there is a block with high certainty in each of the areas A, B, and C, distance information between objects can be obtained using the block. Further, even if there is a portion where distance information cannot be obtained, processing is possible. As an example, if distance information is obtained in one block in each object, it can be used as object distance information. The distance information of other blocks may not be taken.
A block with a high certainty factor is a block having only object pixels overlapping the object area. As shown in FIG. 16, the four blocks in the central portion excluding the block D in the object A are blocks with high certainty.

Next, the main subject extraction method will be described. A shooting focus position is input from the camera parameter input unit 107. Since the focus position of the camera focuses on the main subject, an object having the same distance as the focus position becomes the main subject. For example, when the focus position is on the person of the object A shown in FIG. 18, the person at the same distance becomes the main subject. A region to which the main subject belongs is set as a main subject region, and a region to which the main subject does not belong is set as a non-main subject region.
As shown in FIG. 21, when face detection is performed, an object including a face area is a main subject. When a plurality of faces are detected, an object including the main face area becomes the main subject. The face position is input as a camera parameter from the camera or the like via the camera parameter input unit 107.
The main face area is a rectangular face area detected by face detection. In FIG. 21, a black rectangular frame region overlapping the face region of the object A is a main face region.

(Main subject peripheral erroneous ranging block correction unit 111)
The main subject peripheral erroneous ranging block correction unit 111 serving as a correction distance setting unit uses the main subject region information (main subject region) extracted by the main subject extraction unit 103 to obtain distance information obtained from a camera or the like. Perform correction processing. That is, a new distance is set instead of the obtained block distance for a block having a boundary line between the main subject region and the non-main subject region among the plurality of blocks.

  As shown in FIG. 16, an accurate block distance is not set for the contour block D around the person main object. For example, since the main subject area occupies half of these blocks and the background half occupies the other half, taking an average value of the contrast in the block does not provide an accurate block distance. For this reason, in the contour block D, the pixels in the main subject region have the distance of the main subject as the distance between these pixels (first correction distance). For the pixels in the remaining blocks (pixels in the non-main subject region), the pixel distance (second correction distance) is set by the distance between adjacent blocks belonging to the non-main subject region. By correcting in this way, blurring into the main subject can be eliminated.

  In addition, as shown in FIG. 16, the distance of the object 1 is obtained by averaging all the blocks in the central portion of the object A that is the main subject area (blocks excluding the contour block D in the main subject area). That is, the distance of the main subject. In this way, the same distance is set for all the pixels in the main subject area. However, when all the blocks belonging to the main subject region are the contour block D, the distance (first correction distance) is set based on the pixels in the main subject region included in the contour block D. .

  The distance set for the pixels in the non-main subject area in the contour block D is the distance of the adjacent block belonging to the non-main subject area (second correction distance). If nothing belongs to the main subject area, the distance (second correction distance) is set based on the pixels in the non-main subject area included in the contour block D.

(Error ranging block correction unit 112 for areas other than the main subject)
Next, the erroneous ranging block E in the non-main subject area is removed. In the erroneous ranging block E in the background area C in FIG. 16, the distance difference tends to increase abruptly as compared to the surrounding blocks in which appropriate block distances are set. In this case, interpolation processing is performed using the distance values of the surrounding blocks, and the distance value (block distance) of the erroneous ranging block E is replaced with the corrected block distance. (Error ranging block correction means)

As a specific example, for example, as shown in FIG. 22, the difference in distance information (block distance) from surrounding blocks as a characteristic of the erroneous ranging block E existing in a region other than the main subject (non-main subject region). Is given. Using the difference, the erroneous ranging block E is detected.
As shown in FIG. 23, the erroneous ranging block E determines the difference of the block distances from the surrounding blocks F, G, H, and I, and if it is larger than the set threshold Thre1, to decide.
When it is determined that the erroneous ranging block E is an erroneous ranging, interpolation is performed using the surrounding blocks F, G, H, and I, and the block distance of the erroneous ranging block E is replaced with the corrected block distance.
As the interpolation, for example, a method of performing pixel interpolation by the following equation (6) can be mentioned. Instead of the following equation (6), it may be obtained by other interpolation methods using surrounding blocks.

(Image blur processing unit 104)
Based on the block distance, the first correction distance, the second correction distance, and the correction block distance acquired as described above, the blur processing unit performs blur processing on the image data. When performing blur processing on the background that is non-main image data, different blur processing is performed according to the distance. The blur function acquisition unit 109 acquires blur function data from the blur function DB 110 based on the camera parameters and the shooting distance. When performing blur processing, blur processing is performed for each extracted object using the blur function obtained from the blur function acquisition unit 109. The blur function is a kind of low-pass filter, which is calculated in advance according to the shooting distance and camera parameters. FIG. 24 shows an example of a 5 × 5 Gaussian filter. Each element of the filter, which is a blur function, is calculated based on the shooting distance and camera parameters.

(Blur function DB110)
The blur function DB 110 is a data set of blur functions calculated from design values of various camera parameters such as camera parameters, for example, focal length, aperture size, shooting distance, and pupil shape. If parameters are input, blur function data can be obtained according to the shooting distance.
As shown in FIG. 25, the data structure of the blurring function DB changes the content of the blurring function filter f depending on the focal length L, the F number value, and the shape of the aperture. f is an N × N filter. The value of each element varies depending on camera parameters and shooting distance. The blur database data is calculated in advance from the design value of the camera lens.

(Image composition unit 105, image processing output unit 106)
Finally, the objects subjected to the blurring process by the image composition unit 105 are synthesized, and the image processing output unit 106 outputs an image.

(Digital camera)
Next, a hardware configuration of a digital camera that is an example of an imaging apparatus that performs the above-described image processing will be described. FIG. 26 is a block diagram showing a hardware configuration of the digital camera according to the present embodiment. A flow of a series of processes shown below is shown in FIG.
As shown in FIG. 26, the subject light first enters a CCD (Charge Coupled Device) 3 through the photographing optical system 1. A mechanical shutter 2 is disposed between the photographing optical system 1 and the CCD 3, and incident light on the CCD 3 can be blocked by the mechanical shutter 2. The photographing optical system 1 and the mechanical shutter 2 are driven by a motor driver 6.

  The CCD 3 converts an optical image formed on the imaging surface into an electrical signal and outputs it as analog image data. The image information output from the CCD 3 is subjected to removal of noise components by a CDS (Correlated Double Sampling) circuit 4 and converted to a digital value by an A / D converter 5, and then to an image processing circuit 8. Is output.

  The image processing circuit 8 performs various image processing such as YCrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, and color conversion processing using an SDRAM (Synchronous DRAM) 12 that temporarily stores image data. The white balance process is an image process that adjusts the color density of image information, and the contrast correction process is an image process that adjusts the contrast of image information. The edge enhancement process adjusts the sharpness of image information, and the color conversion process is an image process that adjusts the hue of image information. The image processing circuit 8 displays the image information subjected to signal processing and image processing on a liquid crystal display 16 (hereinafter abbreviated as “LCD 16”).

  The image information subjected to the signal processing and the image processing is recorded in the memory card 14 via the compression / decompression circuit 13. The image compression / decompression circuit 13 compresses the image information output from the image processing circuit 8 according to the instruction acquired from the operation unit 15 and outputs the compressed image information to the memory card 14 and also decompresses the image information read from the memory card 14. This is a circuit for outputting to the image processing circuit 8.

  The timing of the CCD 3, the CDS circuit 4, and the A / D converter 5 is controlled by a CPU (Central Processing Unit) 9 via a timing signal generator 7 that generates a timing signal. Further, the image processing circuit 8, the image compression / decompression circuit 13, and the memory card 14 are also controlled by the CPU 9.

  In the imaging apparatus, the CPU 9 performs various arithmetic processes in accordance with a program, and has a ROM (Read Only Memory) 11 that is a read-only memory storing the program, a work area used in various processes, and various data storage areas. A RAM (Random Access Memory) 10 that is a writable memory is built in, and these are interconnected by a bus line.

  The imaging apparatus first measures distance information of the subject. While moving the lens drive unit 6, the focal position is changed and a plurality of reduced images are input. That is, a plurality of reduced images having different shooting distances are input to the memory SDRAM. After that, the main photographing is performed, and a large size image for background blur processing is input.

  Next, the distance measurement program is called to divide each image into blocks and calculate the sum of contrast values in the same position block. The block with the highest contrast is determined to be in focus at this position, and is set as the image forming position of the frame. The subject distance is calculated from the imaging position. In this way, the photographing distance (block distance) corresponding to all blocks was measured.

Next, an area division / object extraction program is called from the memory to perform object extraction processing. First, an image is divided into small areas using edge information and color information of the image. Next, the objects are extracted by grouping the small areas. When extracting an object, small areas having common attributes are collected from color characteristics and edge characteristics. The measured distance information is also used as region fusion information. Small areas with a short distance are grouped together as object candidates. In this way, the image is separated into a plurality of objects. Here, the object is a meaningful area such as a person, foreground, or background. The shooting distance in each area is different. As shown in FIG. 15, the image is divided into blocks, and each distance is measured by the distance measurement software earlier in each block.
The image is divided into an A object 1, a B object 2, and a background C. As described above, the distances A, B, and C are obtained from blocks belonging to the respective objects.

Next, a main subject is extracted based on camera parameters. When the main subject is extracted, the main subject is extracted according to the distance of the portion matched with the focus of the camera. Alternatively, when photographing a person, the main subject is determined based on the main face area detected by the face area detection.
Further, the contour block D to which the boundary line between the main subject region and the non-main subject region and the erroneous ranging block E in the background are deleted, and the distance information is corrected.
Then, blurring functions of the objects A, B, and C are obtained based on the corrected distance information. When the object A is determined to be the main subject, the blurring process is not performed on A.

  The distance between object B and object C is different. Apply a strong low-pass filter to distant objects, that is, blur it strongly. The blur effect reproduces the same camera blur as the input camera parameters. For example, when a single lens reflex parameter is input, an image with the same background blur as that single lens reflex camera is created.

  The image processing program executed by the digital camera according to the present embodiment has a module configuration including the above-described background blurring function. As actual hardware, a CPU (processor) reads an image processing program from a storage medium. By executing, each unit is loaded on the main storage device, and distance measurement, area division object extraction, blurring processing, image composition, image compression are performed, and an image is generated on the memory card.

(Other embodiment 2)
Another image processing apparatus according to the second embodiment is an example in which an image processing apparatus is used instead of the imaging apparatus. Here, parts different from those of the first embodiment will be described.

In the present embodiment, the recorded image file is input from the imaging device by the image input unit. A process of blurring the background (non-main subject) is performed on the input image data.
FIG. 28 is a block diagram showing a hardware configuration of the image processing apparatus according to the present embodiment. FIG. 29 shows a flow of a series of processes according to the present embodiment.
The image processing apparatus 1000 includes a CPU (Central Processing Unit) 24 that centrally controls each unit. The CPU 24 includes a ROM (Read Only Memory) 22 that is a read-only memory that stores a BIOS and the like. A RAM (Random Access Memory) 21 that stores data so as to be rewritable and functions as a work area of the CPU is connected by a bus to constitute a microcomputer. Further, an HDD 25 in which a control program is stored, a CD-ROM drive 26 that reads a CD (Compact Disc) -ROM, and an I / F 23 that is an interface that manages communication with a printer unit and the like are connected to the bus. .

  A CD-ROM 28 shown in FIG. 28 stores a predetermined control program. The CPU 24 reads the control program stored in the CD-ROM 28 with the CD-ROM drive 26 and installs it in the HDD 25. Thereby, it will be in the state which can perform various processes as mentioned above. The memory card 29 stores image information and the like and is read by the memory card driver 27.

  As a storage medium, not only a CD-ROM and a memory card but also various types of media such as various optical disks such as DVDs, various magnetic disks such as various magneto-optical disks and flexible disks, and semiconductor memories can be used. . Alternatively, the program may be downloaded from a network such as the Internet and installed in the HDD 25. In this case, the storage device that stores the program in the server on the transmission side is also a storage medium. Note that the program may operate on a predetermined OS (Operating System), in which case the OS may execute a part of various processes, or a word processor software, etc. It may be included as part of a group of program files that constitute predetermined application software, OS, or the like.

  The background blurring program executed by the image processing apparatus according to the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the image processing program executed by the image processing apparatus of the present embodiment may be provided or distributed via a network such as the Internet.

  Further, the image processing program according to the present embodiment may be provided by being incorporated in advance in a ROM or the like.

In this embodiment, distance measurement is required when shooting (when an image is recorded), and the measured distance information of each block is recorded in an image file. Camera parameters are recorded in a file. As parameters, the distance information of the focus position and the position information of the main face on the screen when the face is detected are recorded.
The background blur processing program reads the distance information, uses the camera parameters, extracts the blur filter value from the blur database, and performs the blur processing. The processing method is the same as in the first embodiment described above.

  An image processing program executed by the image processing apparatus according to the present embodiment is a file in an installable or executable format, such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or the like. And recorded on a computer-readable recording medium.

(Explanation of symbols in FIGS. 1 to 14)
01 Digital still camera device 02 System control unit 03 Imaging unit 04 Image processing unit 05 Display control unit 06 LCD
07 Recording media interface unit 08 Recording media 09 Hard key interface unit 10 Communication interface unit 11 PC
12 Mode dial 13 Release 14 CCD
15 Lens 62 Release shutter (SW1)
63 Mode dial (SW2)
64 Sub LCD
65 SD card / battery cover 66 Strobe light emitting unit 67 Optical viewfinder 69 Remote control light receiving unit 70 Lens barrel unit 71 AFLED
72 Strobe LED
73 Zoom switch (Wide) (SW3)
74 Zoom switch (remote) (SW4)
75 Self-timer / deletion switch (SW5)
76 Menu switch (SW6)
77 Up / Strobe switch (SW7)
78 Right switch (SW8)
79 Display switch (SW9)
80 Down / Strobe switch (SW10)
81 Left / image confirmation switch (SW11)
82 OK switch (SW12)
83 LCD monitor 84 Power switch (SW13)
100 Image shooting area 101 AF evaluation value area 102 Sky part 103 Curb part 104 Person 105 Townscape 106 Sea 107 AF area 108-113 area 700 AF evaluation value area 701 with division distance information 0 (first stage) AF evaluation value area 702 with 1 (second stage) AF evaluation value area 703 with division distance information of 2 (third stage) AF evaluation value area 704 with division distance information of 3 (fourth stage) One region 705 Pixel 901 Waveform 902 showing a change in AF evaluation value accompanying a scan of a subject at a long distance Waveform 903 showing a change in AF evaluation value accompanying a scan of a subject at a middle distance 903 of a subject at a short distance Waveform 1001 showing a change in AF evaluation value accompanying scanning 1001 Pixel 1002 in the horizontal direction of the target pixel Horizontal direction of the target pixel Pixel 1 of coordinate 1003 Pixel of interest 1004 Pixel of horizontal coordinate of pixel of interest +1 Pixel 1005 of horizontal coordinate of pixel of interest 1102 Pixel of pixel of horizontal direction +2 of coordinate 1101 Distance to sky portion 102 of background 1102 Distance to curb portion 103 of background 1103 104 Distance to the head 1104 Distance to the face of the person 104 1105 Distance to the chest of the person 104 Distance to the townscape 105 and the sea 106 in the background (description of symbols in FIGS. 15 to 29)
A Object 1
B Object 2
C Background D Contour Block E False Distance Measuring Blocks F, G, H, I Surrounding Block 1 Imaging Optical System 2 Mechanical Shutter 3 CCD
4 CDS circuit 5 A / D converter 6 Motor driver 7 Timing signal generator 8 Image processing circuit 9 CPU
10 RAM
11 ROM
12 SDRAM
13 Image Compression / Expansion Circuit 14 Memory 15 Operation Unit 16 LCD
21 RAM (Random Access Memory)
22 ROM (Read Only Memory)
23 I / F
24 CPU (Central Processing Unit)
25 HDD
26 CD-ROM drive 27 Memory card driver 28 CD-ROM
29 memory card 101 image input unit 102 image region dividing unit 103 main subject extraction unit 104 image blurring processing unit 105 image composition processing unit 106 image output unit 107 camera parameter input unit 108 distance information acquisition unit 109 blur function acquisition unit 110 blur function DB
111 Main subject peripheral erroneous ranging block correction unit 112 False ranging block correction unit 201 for areas other than the main subject Reduced image input unit 202 Edge information calculation unit 203 Distance information calculation unit 204 Distance information output unit 205 Lens focus adjustment unit 1000 Image processing apparatus

JP-A-11-266388 JP 2003-37767 A JP 2003-101858 A JP 9-318870 A JP 2009-27298 A

Claims (15)

An image processing apparatus that receives image data including main subject image data and non-main subject image data, and distance information including a distance to the main subject and a distance to the non-main subject, and performs blur processing on the image data Because
Distance dividing means for dividing the distance information into divided distance information having a predetermined number of steps;
A region determination means for determining a region based on luminance and / or chromaticity and outputting a region determination result;
Labeling means for associating the division distance information with the region determination result and outputting label data;
Smoothing means for smoothing the image data by different processing according to the predetermined number of stages and outputting smoothed image data;
Combining processing means for outputting combined image data obtained by combining the image data and the smoothed image data based on the label data,
The image processing apparatus according to claim 1, wherein the labeling unit performs weighting so that the division distance information of the small area is set to 0 in the small area overlapping the boundary of the area including the main subject image data. The distance dividing means divides the distance information into divided distance information consisting of N stages,
2. The image processing according to claim 1, wherein the smoothing unit smoothes the image data with (N−1) different processes and outputs (N−1) different smoothed image data. apparatus.
(However, N is an integer of 2 or more.) A reduction means for reducing the image data;
The image processing apparatus according to claim 1, wherein the region determination unit determines a region in the image data reduced by the reduction unit.   The image processing apparatus according to claim 3, further comprising an enlargement unit that enlarges the label data.   The image processing apparatus according to claim 1, further comprising a composite image data smoothing unit that smoothes the composite image data.   The image processing apparatus according to claim 1, wherein the synthesis processing unit synthesizes the image data and the smoothed image data by weighted addition. Label data smoothing means for outputting smoothed label data obtained by smoothing the label data,
The image processing apparatus according to claim 6, wherein the synthesis processing unit synthesizes the image data and the smoothed image data by weighted addition using the smoothed label data. An image processing method for performing blur processing on image data including input main subject image data and non-main subject image data,
A distance dividing step of dividing the distance information composed of the input distance to the main subject and the distance to the non-main subject into divided distance information having a predetermined number of steps;
A region determination step of determining a region based on luminance and / or chromaticity and outputting a region determination result;
A labeling step of associating the division distance information with the region determination result and outputting label data;
A smoothing step of smoothing the image data with different processing according to the predetermined number of stages and outputting smoothed image data;
A composite processing step of outputting composite image data obtained by combining the image data and the smoothed image data based on the label data,
The image processing method according to claim 1, wherein in the labeling step, weighting is performed so that division distance information of a small area is set to 0 in a small area overlapping a boundary of the area including the main subject image data. The distance dividing step divides the distance information into divided distance information composed of N stages,
9. The image processing according to claim 8, wherein the smoothing step smoothes the image data with (N-1) different processes and outputs smoothed image data with different (N-1). Method.
(However, N is an integer of 2 or more.) A reduction process for reducing the image data;
The image processing method according to claim 8, wherein the region determination step determines a region in the image data reduced in the reduction step.   The image processing method according to claim 10, further comprising an enlarging step of enlarging the label data.   The image processing method according to any one of claims 8 to 11, further comprising a synthetic image data smoothing step for smoothing the synthetic image data.   The image processing method according to any one of claims 8 to 11, wherein in the combining processing step, the image data and the smoothed image data are combined by weighted addition. A label data smoothing step for outputting smoothed label data obtained by smoothing the label data;
The image processing method according to claim 13, wherein the combining processing step combines the image data and the smoothed image data by weighted addition using the smoothed label data.   A digital still camera comprising the image processing apparatus according to claim 1.