JP5282533B2 - Image processing apparatus, imaging apparatus, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and a program that generate a blurred image by image processing using a plurality of images.

  For example, in portrait photography with a camera, a photographic expression that deliberately blurs the foreground or background subject by shallowly opening the aperture to open the main subject is preferred. However, electronic cameras often do not like photographic expressions that take advantage of the above-mentioned blur due to the size of the image sensor (imaging area), the configuration of the diaphragm, and the like.

Therefore, a method for electronically generating an image having a shallow depth of field from a plurality of images captured by an electronic camera has been proposed. As an example, Patent Document 1 discloses a portrait adjustment in which a background region is blurred by acquiring a plurality of images with an electronic camera while changing a focal length or an aperture, and dividing an area using information on a focal length or an aperture at the time of shooting. A technique for generating a blurred image is disclosed.
JP 2003-283902 A

  However, in the above prior art, a blurred image cannot be generated if there is no focal length or aperture information at the time of shooting, and if there is a defect in the area division between the main subject and the background area, There was room for improvement in that the subject was blurred in the image.

  Therefore, an object of the present invention is to provide means for generating a blurred image having a good appearance by image processing using a plurality of images without using focal length or aperture information.

An image processing apparatus according to an aspect includes an image reading unit that acquires a reference image and a plurality of captured images that are captured in a state in which a focal position of a lens is changed with respect to the reference image, and the reference image And an image for extracting a high frequency component for each of the plurality of captured images, combining the high frequency component images indicating the distribution of the extracted high frequency components, and then combining them to generate a reference image having a deeper subject depth than the reference image A processing unit; a feature amount computing unit for obtaining a feature amount of each image based on a luminance component corresponding to each position of the base image and the reference image; and a first feature amount at each position obtained from the base image And a second feature value obtained from the reference image at each position, using the difference between the corresponding positions, a parameter value for setting the parameter value relating to the focus degree of the reference image at each position of the image. And a blurred image generation unit that generates a blurred image by subjecting the reference image to a blurring process that changes a blurring degree at each position of the image according to each parameter value. is there.

  Here, an imaging apparatus including the image processing apparatus according to the one aspect, a program that causes a computer to function as the image processing apparatus according to the one aspect, a program storage medium storing the program, and an image according to the one aspect A representation of the operation in the processing apparatus in the category of the method is also effective as a specific aspect of the present invention.

  According to the present invention, it is possible to generate a blurred image having a good appearance by image processing using a standard image and a reference image without using focal length or aperture information.

<Description of One Embodiment>
FIG. 1 is a block diagram illustrating a schematic configuration of an electronic camera including an image processing apparatus according to an embodiment. The electronic camera 11 includes an imaging optical system 12, a lens driving unit 13, a diaphragm 14, a diaphragm driving unit 15, an imaging element 16, an AFE 17, a CPU 18, a first memory 19 and a second memory 20, and media. It has an I / F 21, a communication I / F 22, a monitor 23, and a release button 24. Here, the lens driving unit 13, the aperture driving unit 15, the AFE 17, the first memory 19, the second memory 20, the media I / F 21, the communication I / F 22, the monitor 23, and the release button 24 are each connected to the CPU 18. .

  The imaging optical system 12 includes a plurality of lenses including a focusing lens. The focal position of the imaging optical system 12 is adjusted in the optical axis direction by the lens driving unit 13. For simplicity, the imaging optical system 12 is illustrated as a single lens in FIG. The diaphragm 14 adjusts the amount of light per unit time incident on the image sensor 16. The aperture of the aperture 14 is adjusted by the aperture drive unit 15 in accordance with an instruction from the CPU 18.

  The imaging element 16 captures a subject image formed by the imaging optical system 12 and generates an image signal of the captured image. The image signal output from the image sensor 16 is input to the AFE 17.

  The AFE 17 is an analog front end circuit that performs analog signal processing on the output of the image sensor 16. The AFE 17 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The digital image signal output from the AFE 17 is input to the CPU 18.

  The CPU 18 is a processor that comprehensively controls the operation of the electronic camera 11. For example, the CPU 18 executes a known AF (autofocus) calculation process using image data captured by the image sensor 16.

  The CPU 18 functions as an image processing unit 25, a feature amount calculation unit 26, and a parameter setting unit 27 by executing a program described later. Here, the image processing unit 25 performs various types of image processing (color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, color conversion processing, etc.) on the captured image after A / D conversion.

  In addition, the image processing unit 25, the feature amount calculation unit 26, and the parameter setting unit 27 generate a blurred image using a plurality of captured images in the background blur mode that is one of the shooting modes. An example of the operation of each unit in the background blur mode will be described later.

  The first memory 19 is composed of a volatile storage medium (SDRAM or the like), and temporarily stores image data in the pre-process and post-process of image processing by the CPU 18. The second memory 20 is configured by a non-volatile storage medium such as a flash memory. The second memory 20 stores a program executed by the CPU 18.

  The media I / F 21 can removably connect a nonvolatile storage medium 28. The media I / F executes writing / reading of image data with respect to the storage medium 28. The storage medium 28 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 28.

  The communication I / F 22 controls data transmission / reception with an external device (such as a computer 31 to be described later) connected via a known wired or wireless communication line in accordance with a predetermined communication standard.

  The monitor 23 reproduces and displays the captured image under the control of the CPU 18. The release button 24 receives from the user an instruction input for starting an AF operation before shooting by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

  Next, an operation example in the background blur mode in the electronic camera according to the embodiment will be described with reference to FIG. Note that the series of processing shown in FIG. 2 is performed on the assumption that the main subject selected by the manual selection operation by the user or automatically selected and selected by the camera itself is in focus. The process starts when the CPU 18 executes a program in response to the full-press operation of 24.

  Step S101: The CPU 18 performs continuous shooting (continuous shooting) for obtaining a plurality of images while changing only the focal position of the imaging optical system 12 from a state in which the main subject is focused (focus bracketing shooting). As a result, the CPU 18 acquires a reference image that is captured in a state where the main subject is in focus, and a plurality of captured images (1 to n frames) that are different in focus from the reference image. An example of the reference image is shown in FIG. 3 (the main subject in the case of FIG. 3 is a “person” located at the center of the screen).

  The reference image and the captured image data output from the image sensor 16 by continuous shooting are stored in the first memory 19 after being subjected to predetermined image processing by the image processing unit 25 via the AFE 17. It becomes.

  Step S102: The image processing unit 25 extracts a high-frequency component for each image for the reference image and each captured image (S101). Thereby, the image processing unit 25 generates a high frequency component image indicating the distribution of the high frequency component in the image for all the images acquired in S101. As an example, the image processing unit 25 in S102 may generate a high-frequency component image from each image by a known unsharp mask process.

  Here, each image acquired in S101 has a different focal position of the imaging optical system 12 at the time of shooting. Therefore, in each high-frequency component image, the location from which the high-frequency component is extracted (in other words, the in-focus location in the image) is different.

  Step S103: The image processing unit 25 synthesizes a plurality of high-frequency component images (generated in S102), and generates a reference image (pan focus image) having a deeper depth of field than the standard image. As an example, the image processing unit 25 in S103 generates a pan focus image by the following processes (1) and (2).

  (1) First, the image processing unit 25 performs image alignment processing over all high-frequency component images by known matching processing between images using edges or corners. This is because during continuous shooting, there is a high possibility that not all images have been shot at the exact same position due to camera shake.

  (2) Next, the image processing unit 25 obtains the pixel value of each pixel of the pan-focus image using the pixel value of the pixel at the corresponding position in each high-frequency component image. For example, the image processing unit 25 may perform integration after averaging a plurality of pixel values at corresponding positions between the images, and use the integrated value as a pixel value in the pan-focus image. Then, the image processing unit 25 generates a pan focus image by repeating the above process for all pixels.

Step S104: The feature amount calculation unit 26 uses the reference image (S101) and the pan focus image (S103) as calculation targets, and obtains feature amounts indicating the dispersion of luminance components at each position of the calculation target image.
As an example, the feature amount calculation unit 26 in S104 specifies a target pixel in the calculation target image, and sets a local region including a plurality of pixels based on the target pixel. Then, the feature amount calculation unit 26 obtains a variance of pixel values corresponding to the luminance component as a feature amount for the pixels included in the local region.

  In S104, if the image data to be calculated is in the YCbCr format, the feature amount calculation unit 26 may obtain the feature amount from the pixel value of the Y component. On the other hand, when the image data to be calculated is in the RGB format, the feature amount calculation unit 26 may obtain the feature amount from any of RGB pixel values, or the pixel of the Y component after color space conversion from RGB to YCbCr. The feature amount may be obtained from the value.

  In the present embodiment, “luminance component dispersion” is used as the feature amount of the image for the following reason.

  First, the reason why luminance information is used is that luminance information is more sensitive (influenced) to human vision than color information. Since human beings are generally more sensitive to luminance differences than color differences, luminance information is used as a reference information amount for appreciating and not causing a sense of incongruity.

  The reason for using variance is that the variance is an amount of information that is sensitive to the focus state of the subject, regardless of the type of feature. (Even if the same image is in focus, (Any feature amount) has a small variance, but if it is out of focus, the variance of the feature amount is smaller than that in the focused state.

  Here, the feature amount calculation unit 26 in S104 calculates the above-described feature amounts for all the pixels of the reference image and the pan focus image. In the following description of the present specification, the feature quantity at each position obtained from the base image is referred to as a first feature quantity, and the feature at each position obtained from the reference image (pan focus image in the example of FIG. 2). Each quantity is referred to as a second feature quantity.

  Step S105: The parameter setting unit 27 uses the first feature value and the second feature value (S104) to set parameter values relating to the focus degree of the reference image for each pixel of the image. Then, the parameter setting unit 27 generates a parameter map indicating the correspondence relationship between the position of each pixel of the reference image and the value of the parameter.

  As an example, the parameter setting unit 27 in S105 obtains a parameter value at a predetermined pixel position by subtracting the first feature quantity from the second feature quantity at the corresponding position on the image. In S105, the values of the above parameters are set for all the pixels of the reference image.

  Here, since the distribution of the high-frequency components of the reference image and the pan-focus image is approximated at the in-focus position on the reference image, the difference between the first feature value and the second feature value at the position is as follows. Get smaller. Therefore, the parameter value obtained in S105 is small at the in-focus position on the reference image.

  On the other hand, in the portion where the subject is blurred on the reference image, the reference image has almost no high-frequency component, but the pan-focus image contains the high-frequency component. The difference from the feature amount becomes large. Therefore, the parameter value obtained in S105 is increased at a portion where the subject is blurred on the reference image. Also, the parameter value changes in accordance with the degree of blur (degree of blur) of the subject on the reference image (the parameter value increases as the degree of blur increases).

  Step S106: The image processing unit 25 performs a blurring process using a smoothing filter on the reference image (S101). Then, the image processing unit 25 in S106 refers to the parameter map (S105), and changes the degree of blurring in the local region of the reference image in a stepwise manner in accordance with the parameter value at each position of the image. .

  As an example, the image processing unit 25 classifies the parameter values into a plurality of classes according to their magnitudes, and performs different blurring processing for each class. When the parameter value is small and belongs to the lowest class at an arbitrary position of the reference image, the image processing unit 25 does not perform the blurring process at the position, or the smoothing filter with the weakest smoothing degree. Apply. On the other hand, when the parameter value belongs to a class one level higher, the image processing unit 25 applies a smoothing filter having a higher degree of smoothing than that of the lowest class. Then, as the parameter value belongs to a higher class, the image processing unit 25 applies a smoothing filter having a higher degree of smoothing to the reference image.

  If the image processing unit 25 is a Gaussian smoothing filter, the degree of smoothing may be changed by adjusting the filter radius. Further, the image processing unit 25 may change the degree of smoothing by using a plurality of smoothing filters (3 × 3, 5 × 5, etc.) having different filter sizes.

  Then, a blurred image in which a background area other than the main subject is blurred is electronically generated by the blurring process. An example of the blurred image corresponding to the reference image in FIG. 3 is shown in FIG. The blurred image data is finally recorded in the storage medium 28 by the CPU 18. At this time, the CPU 18 may also record the reference image data in the storage medium 28. This is the end of the description of FIG.

  The electronic camera 11 according to the above-described embodiment uses the reference image and the pan focus image to obtain a parameter related to the focus degree of the reference image at each position of the image. Then, the electronic camera 11 performs a blurring process in which the blurring degree in the local region of the reference image is changed stepwise based on the above parameters, and the blurring is substantially equivalent to a shooting result with a shallow depth of field. Images are generated electronically.

  Therefore, in one embodiment, it is possible to generate a blurred image in which the subject's blur is natural and looks good without dividing the region of the main subject and the background region. In one embodiment, a blurred image can be generated without using information on the focal length at the time of shooting.

<Description of Modification in One Embodiment>
FIG. 5 is a schematic diagram showing a modification of the background blur mode described in FIG. The series of processing shown in FIG. 5 is also started when the CPU 18 executes a program in response to a full pressing operation of the release button 24 on the assumption that a desired main subject is focused by AF. The configuration of the electronic camera 11 in the example of FIG. 5 is the same as that of the embodiment shown in FIG.

  Step S201: The CPU 18 captures the reference image while focusing on the main subject, and also captures the reference image with the depth of field deepened by reducing the aperture 14 from the state of capturing the reference image. Note that when the reference image is shot in S201, the CPU 18 adjusts the exposure so that the brightness of the reference image is substantially the same as that of the reference image. Note that the data of the standard image and the reference image are stored in the first memory 19 after being subjected to predetermined image processing by the image processing unit 25 via the AFE 17.

  Step S202: The feature amount calculation unit 26 uses the standard image and the reference image (S201) as calculation targets, and obtains feature amounts indicating the variance of the luminance component at each position of the calculation target image. Since the process in S202 is the same as S104 in FIG.

  Step S203: The parameter setting unit 27 uses the first feature value obtained from the reference image and the second feature value obtained from the reference image to set the parameter value relating to the focus degree of the reference image at each pixel of the image. Set each. Then, the parameter setting unit 27 generates a parameter map indicating the correspondence relationship between the position of each pixel of the reference image and the value of the parameter. Since the process in S203 is the same as S105 in FIG.

  Step S204: The image processing unit 25 performs a blurring process with a smoothing filter on the reference image (S201) to generate a blurred image. Since the process in S204 is common to S106 in FIG. This is the end of the description of FIG.

  The electronic camera 11 in the modified example shown in FIG. 5 uses a reference image and a reference image with a deepened depth of field by reducing the aperture 14 and sets parameters relating to the focus degree of the reference image at each position of the image. Seeking in. Also in this case, substantially the same effect as the operation example of the electronic camera 11 described in FIG. 2 can be obtained.

<Description of other embodiments>
FIG. 6 is a block diagram illustrating a configuration example of an image processing apparatus according to another embodiment. In another embodiment, the function of the image processing apparatus is realized by causing the computer 31 to execute an image processing program. Therefore, in the configuration of the other embodiments, substantially the same effects as those of the above-described one embodiment (and its modifications) can be achieved.

  A computer 31 constituting the image processing apparatus includes a data reading unit 32, a storage device 33, a CPU 34, a memory 35, an input / output I / F 36, and a bus 37. The data reading unit 32, the storage device 33, the CPU 34, the memory 35, and the input / output I / F 36 are connected to each other via a bus 37. Furthermore, an input device 38 (keyboard, pointing device, etc.) and a monitor 39 are connected to the computer 31 via an input / output I / F 36. The input / output I / F 36 accepts various inputs from the input device 38 and outputs display data to the monitor 39.

  The data reading unit 32 is used when reading captured image data and an image processing program from the outside. For example, the data reading unit 32 is a reading device (such as a reading device for an optical disk, a magnetic disk, or a magneto-optical disk) that acquires data from a removable storage medium, or an external device (electronic camera) in accordance with a known communication standard. 11) and the like (communication device such as USB interface, wired or wireless LAN module).

  The storage device 33 stores the image processing program and various data necessary for executing the program. The storage device 33 can also record captured image data acquired from the data reading unit 32. Note that the storage device 33 in another embodiment is configured by a hard disk, a nonvolatile semiconductor memory, or the like.

  The CPU 34 is a processor that comprehensively controls the operation of each part of the computer 31. The CPU 34 functions as the image processing unit 25, the feature amount calculation unit 26, and the parameter setting unit 27, respectively, by executing the image processing program. The memory 35 temporarily stores the calculation result of the image processing program. The memory 35 is composed of, for example, a volatile SDRAM.

  Here, when the blurred image is generated by the computer 31 illustrated in FIG. 6, the user preliminarily captures the reference image and a plurality of captured images having different focal positions (or the reference image captured by narrowing the aperture 14 from the shooting state of the reference image). ) With the electronic camera 11. Then, the computer 31 may acquire the data of each image from the data reading unit 32 and execute the processing after S102 in FIG. 2 or the processing after S202 in FIG.

  Note that the determination of the reference image may be performed by the computer 31 based on a user's designation operation. Alternatively, when the electronic camera 11 adds auxiliary data indicating the reference image to the header of the image file at the time of shooting, the computer 31 can determine the reference image based on the presence or absence of the auxiliary data.

<Supplementary items of the embodiment>
(1) The image processing apparatus of the present invention is not limited to the examples of the electronic camera 11 and the computer 31 of the above embodiment, and can be widely applied to a mobile phone with a camera, an image viewer, and the like.

  (2) In each of the above-described embodiments, an example has been described in which the operations of the image processing unit 25, the feature amount calculation unit 26, and the parameter setting unit 27 are implemented by software using a program. Of course, it may be realized.

  (3) In the above-described one embodiment and its modification, the example in which the blurred image is generated when the electronic camera 11 captures the image has been described. However, also for the electronic camera 11 described above, a pre-captured image may be read from the storage medium 28 in a post-processing step to generate a blurred image.

  (4) In the above embodiment, an example has been described in which parameter values are obtained for all the pixels of the reference image. However, the electronic camera 11 and the computer 31 may use the reference image as a small block having a size of about 10 pixels × 10 pixels. The parameter value may be obtained for each block by dividing the area into blocks.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

1 is a block diagram showing a schematic configuration of an electronic camera according to an embodiment. FIG. 6 is a schematic diagram illustrating an operation example in the background blur mode according to the embodiment. The figure which shows an example of a reference image The figure which shows the example of the blurring image corresponding to the reference | standard image of FIG. Schematic diagram showing a variation of the background blur mode of FIG. The block diagram which shows the structural example of the image processing apparatus which concerns on other embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Imaging optical system, 13 ... Lens drive part, 14 ... Aperture, 15 ... Aperture drive part, 16 ... Imaging element, 18 ... CPU, 25 ... Image processing part, 26 ... Feature-value calculating part, 27 ... Parameter setting unit, 31 ... Computer, 32 ... Data reading unit, 33 ... Storage device, 34 ... CPU

Claims (10)

An image reading unit for acquiring a reference image and a plurality of captured images captured in a state in which a focal position of a lens is changed with respect to the reference image;
A high-frequency component is extracted for each of the reference image and the plurality of captured images, and a high-frequency component image indicating the distribution of the extracted high-frequency components is aligned and synthesized to generate a reference image having a deeper subject depth than the reference image. An image processing unit to be generated;
A feature amount computing unit for obtaining a feature amount of each image based on a luminance component corresponding to each position of the standard image and the reference image ;
Using the difference between corresponding positions of the first feature value at each position obtained from the reference image and the second feature value at each position obtained from the reference image, the degree of focus of the reference image A parameter setting unit for setting the value of the parameter at each position of the image;
A blur image generating unit that generates a blur image by performing blur processing on the reference image to change the blur degree at each position of the image according to the value of each of the parameters;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image processing unit performs image alignment processing over all the high-frequency image component images by matching processing between the recording standard image and the reference image, and then a plurality of frequency component images at corresponding positions. An image processing apparatus , wherein the reference image is generated by repeating a process of setting the pixel value of the reference image as a pixel value of the reference image after adding and adjusting the weighted average of the pixel values . The image processing apparatus according to claim 1 or 2,
The feature amount calculation unit specifies a target pixel in the standard image and the reference image, sets a local region including a plurality of pixels based on the specified target pixel, and includes pixels included in the set local region An image processing apparatus characterized by obtaining a variance of pixel values corresponding to a luminance component as a feature amount . The image processing apparatus according to claim 1 or 2,
The image processing apparatus according to claim 1, wherein the image data of the reference image and the reference image is obtained from a pixel value of a Y component when the image data of the standard image and the reference image is in a YCbCr format . The image processing apparatus according to claim 1 or 2,
When the image data of the standard image and the reference image is in RGB format, the feature amount calculation unit performs color conversion from a pixel value of any image data of RGB image data or from RGB image data to YCbCr image data. An image processing apparatus characterized by obtaining a feature amount from a pixel value of Y component image data in the converted YCbCr image data after performing spatial conversion . The image processing apparatus according to any one of claims 1 to 5,
  The parameter setting unit obtains the value of the parameter at all positions by subtracting the first feature quantity from the second feature quantity at a corresponding position on the image. . The image processing apparatus according to any one of claims 1 to 6,
  The image generation apparatus according to claim 1, wherein the image generation unit classifies the parameter values into a plurality of classes according to the size thereof, and performs different blurring processing for each class. The image processing apparatus according to claim 7.
  When the parameter value is small and belongs to the lowest class at an arbitrary position of the reference image, the image generation unit does not perform the blurring process at the position or performs smoothing with the weakest degree of smoothing. If the parameter value belongs to a class one level higher, a smoothing filter having a higher degree of smoothing than that of the lowest class is applied, and the parameter value is more An image processing apparatus that applies a smoothing filter having a higher degree of smoothing to a reference image as belonging to a higher class. An imaging unit that images a subject and generates an image;
  The image processing apparatus according to any one of claims 1 to 8,
  An imaging apparatus comprising: An image reading unit for acquiring a reference image and a plurality of captured images captured in a state in which a focal position of a lens is changed with respect to the reference image;
  A high-frequency component is extracted for each of the reference image and the plurality of captured images, and a high-frequency component image indicating the distribution of the extracted high-frequency components is aligned and synthesized to generate a reference image having a deeper subject depth than the reference image. An image processing unit to be generated;
  A feature amount calculation process for obtaining a feature amount of each image based on a luminance component corresponding to each position of the reference image and the reference image;
  The degree of focus of the image is indicated by using the difference between the corresponding positions of the first feature value at each position obtained from the reference image and the second feature value at each position obtained from the reference image. A parameter setting process for setting the parameter value at each position of the image;
  A blur image generation process for generating a blur image by performing blur processing on the reference image to change the blur degree at each position of the image according to the value of each of the parameters;
  A program that causes a computer to execute.

Images