JP5144724B2 - Imaging apparatus, image processing apparatus, imaging method, and image processing method - Google Patents

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, an imaging method, and an image processing method, and in particular, an imaging apparatus, an image processing apparatus, an imaging method, and an imaging apparatus that obtain a blur-emphasized image by combining a plurality of images having different focal positions. The present invention relates to an image processing method.

  Conventionally, in photography, a method has been used in which only the main subject is focused by intentionally blurring other parts other than the main subject by setting the aperture to the open side and reducing the depth of field. It was. However, this type of photography is possible only for large cameras, such as single-lens reflex cameras, where the aperture can be adjusted manually, and compact cameras, etc., have a small image sensor, so the aperture is automatically controlled to the open side. Even so, it was difficult to blur areas other than the main subject.

  In order to solve such a problem, a technique for obtaining a blur-enhanced image by performing image processing on captured image data is known. Patent Document 1 discloses a digital camera that detects a non-focused area from captured image data and performs a blurring process on the non-focused area by digital signal processing. According to the digital camera of Patent Document 1, even when an image with a deep depth of field is taken, other parts other than the main subject are the same as those taken with a shallow depth of field by the blurring process. An intentionally blurred image can be obtained.

JP 2006-140594 A

  However, in the digital camera described in Patent Document 1, since the out-of-focus area is determined by light intensity change or the like without considering the frequency characteristics of the subject, whether the subject is originally a low-frequency subject or not by focusing. It was impossible to distinguish whether the frequency was low, and it was not possible to determine a good out-of-focus area. For this reason, there is a drawback in that the area to be blurred cannot be appropriately blurred.

  The present invention has been made in view of such circumstances, and in a compact camera or the like where it is difficult to capture an image with a shallow depth of field, an intended blur-enhanced image obtained by appropriately blurring a region to be blurred is provided. An object is to provide an imaging device, an image processing device, an imaging method, and an image processing method that can be obtained.

In order to achieve the above object, an imaging apparatus according to the present invention includes an automatic focusing position determination unit that determines a focusing position of a main subject based on image data, and discretely sets a focal position with a predetermined amount of movement. A focus bracket photographing unit that obtains a plurality of images by moving and sequentially capturing images, and a focus position of the main subject in which the moving focus position is determined by the automatic focus position determination unit and a focus before and after the focus position Focus position control means for controlling to include a position , wherein the focus position is controlled so as to include a plurality of focus positions before the focus position of the main subject and to include a plurality of focus positions after the main subject. the focus bra comprising a focus position control means, the image captured at the focal position of the front and rear focus position of the photographed image and the main subject in-focus position of the main subject Obtained by the sharpness calculating means for calculating the sharpness of the pixels at the corresponding coordinates between the plurality of images photographed by the image photographing means, the image synthesizing means for synthesizing the plurality of images, and the image synthesizing means. A recording unit configured to record an image on a recording medium; and a blur enhancement level setting unit configured to set a blur enhancement level , wherein the image synthesis unit is configured to focus the main subject at pixels having corresponding coordinates between the plurality of images. When the sharpness of the pixel photographed at the position is maximum, the pixel is selected, and when the sharpness of the pixel photographed at the focal position before the in-focus position of the main subject is maximum, the main subject is selected. A pixel corresponding to the degree of blur enhancement is selected from a plurality of images photographed at a focal position after the in-focus position, and the sharpness of the pixel photographed at the focal position after the in-focus position of the main subject If is the largest Selects the pixels corresponding plurality of images than the focus position of the object taken with the focal position before the blur enhancement degree, characterized in that the image synthesis.

  Thereby, a blur-emphasized image can be obtained.

In addition, it is possible to blur an area to be blurred with an appropriate blur amount, and an intended blur-enhanced image can be obtained. Furthermore, a blur-enhanced image with a blur amount according to the user's preference can be taken.

  Feature point extraction means for extracting a feature point of a subject in an image photographed at the focus position of the main subject, and the feature point in an image photographed at a focal position before and after the focus position of the main subject. Corresponding point detection means for detecting corresponding points of the corresponding subject, and images taken at focal positions before and after the in-focus position of the main subject so that the coordinates of the corresponding points coincide with the coordinates of the corresponding feature points, respectively. An image deformation means for deforming an image, wherein the sharpness calculation means is photographed at an image photographed at a focus position of the main subject and at a focal position before and after the focus position of the deformed main subject. It is preferable to calculate the sharpness of the pixel at the corresponding coordinates between a plurality of images.

  It is preferable that the focal position control means controls the moving focal position based on at least one of a diaphragm at the time of photographing and a focal length at the time of photographing.

  This makes it possible to perform focus bracket shooting according to the depth of field.

  A blur enhancement level setting unit configured to set a blur enhancement level, wherein the focus position control unit performs control so as to include a plurality of focus positions before the focus position of the main subject, and sets a focus position after the main subject. A plurality of images, and the image composition means has a maximum sharpness of a pixel photographed at a focal position before the in-focus position of the main subject in pixels corresponding to coordinates between the plurality of images. Selects a pixel corresponding to the degree of blur enhancement from a plurality of images taken at a focal position after the focus position of the main subject, and is taken at a focus position after the focus position of the main subject. If the sharpness of the pixel is maximum, the image may be synthesized by selecting pixels corresponding to the blur enhancement degree from a plurality of images taken at a focal position before the focus position of the main subject.

  As a result, a blur-enhanced image with a blur amount according to the user's preference can be taken.

In order to achieve the above object, the image processing apparatus according to claim 4 , the image photographed at a focus position of the main subject and a focus position before the focus position before and after the focus position of the main subject. An input means for inputting a plurality of images of the same scene captured at different focal positions, including a plurality of images and a plurality of images captured at a focal position after the focus position of the main subject ; Sharpness calculation means for calculating the sharpness of pixels at corresponding coordinates between images, image composition means for combining the plurality of images, output means for outputting an image obtained by the image composition means , and blur enhancement A blur enhancement degree setting means for setting a degree , wherein the image composition means has a maximum sharpness of a pixel photographed at a focus position of the main subject among pixels of corresponding coordinates between the plurality of images. Haso Selects a pixel, the main object focus of pixels taken at a focal position before the position when the sharpness of the largest multiple photographed at the focal position of after the focus position of the main subject A pixel corresponding to the degree of blur enhancement is selected from an image, and when the sharpness of a pixel photographed at a focal position after the in-focus position of the main subject is maximum, the in-focus position of the main subject is A pixel corresponding to the degree of blur enhancement is selected from a plurality of images photographed at a focal position, and the images are synthesized.

In order to achieve the above object, the imaging method according to claim 5 includes an automatic in-focus position determining step of determining an in-focus position of a main subject based on image data, and discrete focusing with a predetermined amount of movement. A focus bracket imaging step of obtaining a plurality of images by moving the position and sequentially capturing images; and a focus position of the main subject in which the moving focus position is determined by the automatic focus position determination step; A focus position control step for controlling to include a focal position before and after the in-focus position of the subject, wherein the focus position is controlled so as to include a plurality of focal positions before the in-focus position of the main subject and after the main subject. a focus position control process for controlling so as to include a plurality of focal positions, the shot at the focal position of the front and rear focus position of the photographed image and the main subject in-focus position of the main subject image A sharpness calculation step of calculating the sharpness of pixels of corresponding coordinates between a plurality of images photographed by the focus bracket photographing step, an image composition step of compositing the plurality of images, and the image composition step A recording step of recording the obtained image on a recording medium, and a blur enhancement level setting step of setting a blur enhancement level , wherein the image synthesis step includes the main subject at pixels of corresponding coordinates between the plurality of images. If the sharpness of the pixel photographed at the in-focus position is the maximum, the pixel is selected.If the sharpness of the pixel photographed at the focal position before the in-focus position of the main subject is the maximum, the pixel is selected. Pixels corresponding to the degree of blur enhancement are selected from a plurality of images shot at a focal position after the focus position of the main subject, and pixels shot at a focus position after the focus position of the main subject of Sharpness is the case of the maximum, characterized in that the selected pixels corresponding to the blur enhancement degree from a plurality of images photographed at the focal position before the focus position of the main object image synthesis.

In order to achieve the object, an image processing method according to claim 6 , comprising: an image photographed at a focus position of a main subject; and a plurality of images photographed at a focus position before the focus position of the main subject. An input step for inputting a plurality of images of the same scene photographed at different focal positions including a plurality of images photographed at a focal position after the focus position of the main subject, and a correspondence between the plurality of images A sharpness calculation step of calculating the sharpness of the pixel of coordinates, an image synthesis step of synthesizing the plurality of images, an output step of outputting an image obtained by the image synthesis step, and a blur setting degree An emphasis degree setting step, and the image composition step selects a pixel when the sharpness of a pixel photographed at the in-focus position of the main subject is maximum among pixels of corresponding coordinates between the plurality of images. And Serial The blur enhancement from the photographed plurality of image at the focus position after even if the sharpness is maximum than the in-focus position of the main subject of the pixels is also taken at the focus position of the pre-focus position of the main subject Select the pixel according to the degree, and if the sharpness of the pixel photographed at the focal position after the focus position of the main subject is the maximum, photograph at the focus position before the focus position of the main subject A pixel corresponding to the degree of blur enhancement is selected from the plurality of images, and image synthesis is performed.

  According to the present invention, it is possible to obtain a blur-enhanced image that is more natural than normal shooting. Thereby, using a camera with a small imaging device such as a compact camera, an imaging device that obtains a shallow, blurred image with a shallow depth of field such as a single-lens reflex camera that is photographed with a large imaging device, An image processing apparatus, an imaging method, and an image processing method can be provided.

FIG. 1 is a block diagram showing an example of the internal configuration of the digital camera 1 according to the first embodiment of the present invention. FIG. 2 is a flowchart showing a community registration operation. FIG. 3 is a diagram illustrating an image obtained from the focus bracket shooting. FIG. 4 is a flowchart showing the operation of detecting corresponding points of the digital camera 1. FIG. 5 is a diagram illustrating image deformation. FIG. 6 is a flowchart showing the operation of the image composition process. FIG. 7 is a diagram illustrating a filter matrix of a Laplacian filter. FIG. 8 is a flowchart showing the operation in the blur enhancement shooting mode. FIG. 9 is a graph showing each focus position and the AF evaluation value at each focus position. FIG. 10 is a flowchart of the operation of the image composition process in the second embodiment. FIG. 11 is a graph showing the sharpness of an image captured at the focus position B of the main subject and images captured at A and C, which are focal positions before and after the focus position. FIG. 12 is a block diagram showing an example of the internal configuration of the digital camera 1 according to the third embodiment of the present invention. FIG. 13 is a flowchart illustrating the operation in the blur-enhanced shooting mode according to the third embodiment. FIG. 14 is a graph showing each focus position and the AF evaluation value at each focus position. FIG. 15 is a graph showing the relationship between the aperture, the focal distance, and the distance d between the focal positions. FIG. 16 is a flowchart showing the operation of the blur amount calculation process. FIG. 17 is a flowchart showing the operation of the blurring process. FIG. 18 is a graph showing the relationship between the absolute value | k | of the blur amount and the σ parameter of the Gaussian filter.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing an example of the internal configuration of the digital camera 1 according to the first embodiment of the present invention.

  The CPU 24 performs overall control of each circuit in the digital camera 1 based on the input of the operation unit 23 including the shutter switch, and executes processing according to the camera control program.

Control of each circuit of the CPU 24 is performed via an address bus 25 and a data bus 26. The CPU 24 exchanges necessary data with the main memory 28. The main memory 28 is divided into a ROM area and a RAM area. The ROM area includes a camera control program, an opening image at startup, an ending image at stop, a menu image used for operating the digital camera 1, and the like. GUI image, screen saver image, progress display image during processing (such as an hourglass image whose scale changes), key operation sound (shutter sound, etc.), sound data indicating warning sound, error sound, etc. Is recorded.

  When the digital camera 1 is turned on, the CPU 24 detects this, displays the opening image stored in the ROM area of the main memory 28 for a certain period on the display unit 35, and then enters the shooting standby state in the shooting mode. . In the photographing standby state, the CPU 24 displays a moving image (through image) on the display unit 35.

  The user (photographer) performs framing while viewing the through image displayed on the display unit 35, confirms the subject to be photographed, confirms the image after photographing, and sets photographing conditions.

  When a shutter switch (not shown) of the operation unit 23 is pressed during the shooting standby state, the CPU 24 determines the in-focus position based on the AF evaluation value calculated by the integrating unit 31 and moves the lens 11 through the lens driving unit 16. Drive and focus control, drive aperture 12 through aperture drive unit 17 to perform exposure control, and light receiving surface of solid-state image sensor 15 through lens 11, aperture 12, Ir cut filter 13, and optical low-pass filter 14 A subject image is formed on the top. Here, if necessary, the strobe 19 is caused to emit light as auxiliary photographing light. At this time, the light receiving unit 20 controls the light emission amount of the strobe 19. The subject image formed on the light receiving surface of the solid-state image sensor 15 is converted into a signal charge of an amount corresponding to the amount of light, and this signal charge is analog based on a drive pulse given from the image sensor drive unit 18 according to a command from the CPU 24. The signal is sent to the signal processing unit 21, where it is subjected to correlated double sampling processing, amplified, and added to the A / D converter 22.

  The dot sequential R, G, B signals converted into digital signals by the A / D converter 22 are stored in the RAM area of the main memory 28 via the main memory control unit 27.

  The digital signal processing unit 29 reads the R, G, and B raw data stored in the RAM area of the main memory 28 and applies a digital gain corresponding to the light source type to the white data to adjust the white balance. (Tone conversion) processing, sharpness processing, etc. are performed to generate R, G, B signals. Further, YC signal processing is performed to generate a luminance signal Y and chroma signals Cr and Cb (YC signals) and store them again in the RAM area of the main memory 28.

  The YC signal stored in the RAM area of the main memory 28 as described above is compressed into a predetermined format by the compression / decompression processing unit 30 and is then detachably attached to the digital camera 1 via the memory control unit 32. It is recorded on the medium 33.

  When the playback mode is selected by operating the various operation units 23, the image file of the last frame recorded on the recording medium is read via the memory control unit 32. The compressed data of the read image file is expanded into an uncompressed YC signal by the compression / decompression processor 30. The expanded YC signal is converted into a display signal format and output to the display unit 35. Thereby, the image of the last frame recorded on the recording medium is displayed on the display unit 35.

  Next, omnifocal image shooting in the digital camera 1 will be described. FIG. 2 is a flowchart showing the operation of an omnifocal image capturing mode for capturing an omnifocal image. In the omnifocal image shooting mode of the digital camera 1 according to the present invention, focus bracket shooting is performed, the corresponding points of the captured images are detected, the captured images are deformed so that the positions of the corresponding points coincide, and the deformed images are obtained. Are combined to obtain an omnifocal image.

  By operating a mode switch (not shown) of the operation unit 23, the omnifocal image capturing mode is set. When the shutter switch of the operation unit 23 is further pressed, the CPU 24 sets the focal position based on these operations (step S21), photographing is performed (step S22). Next, the number of shots is determined (step S23). If it is less than the predetermined number of times, a different focal position is set (step S21), and further photographing is performed (step S22). In this way, photographing at different focal positions is repeated, and photographing is performed a predetermined number of times (focus bracket photographing). Each captured image is stored in the RAM area of the main memory 28.

  Here, the predetermined number of times may be a predetermined number in the digital camera 1 or may be set by the user. Alternatively, the depth of field determined by the shooting conditions may be determined and determined according to the depth of field. In order to obtain an omnifocal image, it is necessary that all the subjects in the image are in focus in any one of the images. Therefore, when the depth of field is shallow, it is necessary to reduce the amount of movement of the focal position and increase the number of shots. In addition, when the depth of field is deep, by increasing the amount of movement of the focal position and reducing the number of shots, it is possible to prevent unnecessary consumption of memory by taking a large number of shots and increasing the processing load. Can do.

  Further, regarding the movement of the focal position, the focal position may be moved from a position where the focal distance is close to a far position, may be moved from a far position to a close position, or may be moved randomly.

  When the predetermined number of times of shooting is completed, corresponding points in each shot image are detected (step S24).

  Here, detection of corresponding points in the image will be described. Corresponding point detection is to obtain the positional relationship of corresponding points between a plurality of images. FIG. 3 is a diagram illustrating an image obtained from the focus bracket shooting. Corresponding point detection unit 36 extracts a feature point from the reference image shown in FIG. 3A and tracks to which position in the tracking image shown in FIG. Perform point detection. FIG. 4 is a flowchart showing the operation of detecting corresponding points of the digital camera 1.

  First, the CPU 24 inputs a plurality of images obtained by focus bracket shooting to the corresponding point detection unit 36 (step S41). Next, the corresponding point detection unit 36 selects a reference image from the plurality of images (step S42), and extracts feature points from the reference image (step S43). The reference image may be selected based on a predetermined method in the digital camera 1 or may be selected by the user. Here, the first photographed image is used as the reference image. Various feature point extraction methods have been proposed. Here, when the standard deviation of luminance in a predetermined area centered on a certain point is a predetermined value or more, that point is extracted as a feature point. Circles in FIG. 3A indicate feature points in the image of FIG. As the number of feature points increases, the subsequent processing can be performed more accurately. However, since the processing load increases as the number of feature points increases, the number of feature points may be determined as appropriate from the hardware performance.

  Next, a tracking image is selected (step S44). Of the plurality of images input in step S41, an image other than the reference image is selected.

  In the selected tracking image, one of the feature points extracted from the reference image is selected (step S45), and the position to which the feature point has moved is tracked (step S46). Various tracking methods have also been proposed. Here, tracking is performed using a method (block matching method) that finds coordinates that minimize the cross-correlation coefficient within a predetermined area centered on that point. Do. Since focus bracket photography is performed in the present invention, the direction and amount of movement of the corresponding point as the focal position moves can be predicted. The movement of the corresponding point by the movement of the focal position is a movement from the center of the screen in the radial direction or in the opposite direction, and this can be predicted from the movement direction of the focal position. The amount of movement can also be predicted from the amount of movement of the focal position. Therefore, by detecting the corresponding points in consideration of these, the corresponding point detection performance can be improved in both accuracy and speed.

  When the tracking of feature points is completed, it is determined whether or not processing has been performed for all feature points (step S47). If there is a feature point that has not been processed, the feature point is selected (step S45), and tracking is performed (step S46). In this way, tracking is performed for all feature points.

  If processing has been performed for all feature points, it is next determined whether or not this processing has been performed for all tracking images (step S47). As described above, this process is performed for images other than the reference image. If there is a tracking image that has not been processed, the same processing is performed on the tracking image. When the process is finished for all the tracking images, the corresponding point detection process is finished.

  When the corresponding point detection process is completed, the image is then deformed (step S25). In this image deformation process, the image deformation unit 37 deforms the tracking image so that the corresponding point of each tracking image obtained by the corresponding point detection process matches the position of the corresponding point of the reference image as much as possible. FIG. 5 is a diagram illustrating image deformation. When the corresponding points are translated with respect to the reference image shown in FIG. 5A, translation is performed. Further, as shown in FIG. 5C, when each corresponding point is rotationally moved, the rotational movement is performed. Further, as shown in FIG. 5D, when each corresponding point is enlarged (reduced), reduction (enlargement) is performed. For these image deformations, the movement vector may be determined so that the total sum of the distances between a plurality of sets of corresponding points is minimized. These parallel movement, rotation, and enlargement / reduction processing are performed by affine transformation.

  Further, as shown in FIG. 5E, when a complicated movement occurs between the reference image and the tracking image, warping is performed in order to match all corresponding points more accurately. In the warping, a movement vector that completely matches a set of all corresponding points is selected, and surrounding points are also obtained by interpolation.

  In this way, image synthesis is performed on a plurality of images in which the positions of corresponding points are matched as much as possible (step S26). FIG. 6 is a flowchart showing the operation of the image composition process. In the image composition process, an image having the highest sharpness in a plurality of images, that is, an image having a high sharpness at any position in the screen as a final image by selecting and synthesizing the pixel values of the most focused image, That is, an omnifocal image is obtained.

  First, the image composition unit 38 selects a pixel to be processed (step S61). The order of selecting the pixels may be performed in order from the edge of the screen, and other orders may be used. Next, an image is selected (step S62), and the sharpness of the pixel selected in step S61 in the selected image is calculated (step S63).

  The sharpness is calculated by calculating the absolute value of the output value by Laplacian filter processing. FIG. 7 is a diagram illustrating a filter matrix of a Laplacian filter. Edge detection can be performed by performing Laplacian filter processing, and the absolute value of the output value indicates the sharpness. Regarding the blur and sharpness of an image, there is a relationship in which a pixel having a smaller blur of the image has a higher sharpness and a sharper the lower the blur is. The kernel of the Laplacian filter is not limited to this example, and a sharpness calculation filter other than the Laplacian filter may be used.

  Next, it is determined whether or not the sharpness calculation has been completed for all images (step S64). If there is an image whose sharpness is not calculated, the image is selected again (step S62), and the sharpness of the pixel selected in step S61 in the selected image is calculated (step S63). In this way, the sharpness of the pixel selected in step S61 is calculated for all images.

  Next, the pixel value of the image with the maximum sharpness is output (step S65). The absolute value of the output value in the Laplacian filter processing of the pixel selected in step S61 of each image calculated in step S63 is compared, and the pixel value of the image that is the maximum value is used as the output pixel. Alternatively, the weighted average may be calculated by increasing the weight for the pixel value of the image with high sharpness, and this may be output.

  The above processing is performed for all pixels. It is determined whether or not processing has been performed for all pixels, and when the processing has been completed for all pixels, the image composition processing ends.

  When the image composition process is completed, the CPU 24 records the output image of the image composition unit 38 on the recording medium 33 via the memory control unit 32, and the omnifocal image capturing is completed (step S27).

  The output image of the image composition unit 38 may be displayed on the display unit 35 via the display control unit 34 without being stored in the recording medium 33, or displayed on an external monitor using an external interface (not shown). Or output as a printed matter to a printer or the like.

  In this way, an omnifocal image can be obtained even when there is subject blur or camera shake.

  The feature point extraction method, the feature point tracking method, the image deformation method, and the pixel sharpness calculation method are not limited to the method of the present embodiment, and any method may be used.

<Second Embodiment>
Next, blur enhanced shooting in the digital camera 1 will be described. FIG. 8 is a flowchart showing an operation in a blur-enhanced shooting mode for shooting a blur-enhanced image. The parts common to the flowchart of FIG. 2 are denoted by the same reference numerals, and detailed description thereof is omitted.

  In the blur-enhanced image shooting mode of the digital camera 1 according to the present invention, focus bracket shooting including shooting at the in-focus position of the main subject and the front and rear focus positions is performed by autofocus, and corresponding points of each captured image are detected. Then, each captured image is deformed so that the positions of the corresponding points coincide with each other, and the blurred image is obtained by combining the deformed images.

  First, the blur-enhanced image shooting mode is set by operating the mode switch of the operation unit 23. Next, when the shutter button in the operation unit 23 is pressed, automatic focus position determination (AF) is first performed on the main subject (step S81). This AF may be the same as that performed during normal shooting, and the focal position of the lens is driven prior to shooting, and an AF evaluation value such as an output value of a bandpass filter is calculated by the integrating unit 31 each time. Then, the CPU determines that the focus position with the highest AF evaluation value is the in-focus position.

  In focus bracket shooting in the blur-enhanced image shooting mode, shooting is performed by moving the focus position to include the AF focus position of the main subject and the focus positions before and after the AF focus position of the main subject. FIG. 9 is a graph in which each focal position of the digital camera 1 is represented on the horizontal axis, and the AF evaluation value for the main subject at each focal position is represented on the vertical axis. The focus position with the highest AF evaluation value is the focus position of the main subject. If the focus position of the main subject is B, in the focus bracket shooting of the present embodiment, the focus position B is Three focal points A and C that are separated by a predetermined distance d are taken as focal positions at the time of photographing. This d may be a fixed value stored in advance in the digital camera 1 or may be set by the user. Further, the depth of field may be determined and varied depending on the depth of field. Since the degree of blur differs depending on the depth of field, it is possible to obtain an image with an appropriate degree of blur by changing the value of d depending on the depth of field.

  At the focal position determined in this way, focus bracket shooting is performed in the same manner as in the omnifocal image shooting mode of the first embodiment (steps S21 to S23). Each captured image is stored in the RAM area of the main memory 28.

  When focus bracket photography is completed, corresponding points in each photographed image are detected (step S24). Here, it is assumed that the reference image is an image taken at the in-focus position B of the main subject. Corresponding point detection is performed in the same manner as in the first embodiment.

  When the corresponding point detection process is completed, the image is then deformed (step S25). The image deformation is performed in the same manner as in the first embodiment.

  When the image transformation process is completed, image composition is performed next. FIG. 10 is a flowchart of the operation of the image composition process in the second embodiment. In addition, the same code | symbol is attached | subjected to the part which is common in the flowchart of FIG. 6, and the detailed description is abbreviate | omitted. In the blur-enhanced image shooting mode of the digital camera 1 according to the second embodiment of the present invention, when the reference image has the highest sharpness in a plurality of images in the image composition process, a pixel of the reference image is selected, and the reference image If the other image has the highest sharpness, blur enhancement is selected by selecting the pixel of the image taken at the focal position where the sharpness of the pixel is symmetric with respect to the focus position of the main subject. Get an image.

  Similar to the first embodiment, the image composition unit 38 selects a pixel to be processed (step S61), and calculates the sharpness of each image of the selected pixel (steps S62 to S64). FIG. 11 is a graph showing the relationship between the sharpness of an image shot at the focus position B of the main subject and images shot at the focus positions A and C before and after the focus position. When three images are captured by focus bracket imaging, the relationship between the sharpness of images captured at the focal positions of A, B, and C is as shown in FIG. 11 when C shown in FIG. It can be classified into three types when B shown in (b) is maximum and when A shown in FIG. 11 (c) is maximum.

  When the sharpness of the selected pixel in all the images is calculated, based on the calculated sharpness, whether or not the sharpness of the reference image is the maximum, that is, the sharpness is in the relationship shown in FIG. It is determined whether or not there is (step S101). If the sharpness of the reference image is maximum, the pixel value of the reference image is output (step S102). When the sharpness of the reference image is not the maximum, whether the sharpness of the image farther from the in-focus position of the main subject is the maximum, that is, whether the sharpness has the relationship shown in FIG. Is determined (step S103). When the sharpness of the image farther from the focus position of the main subject is maximum, the pixel value of the image closer to the focus position of the main subject is output (step S104). On the contrary, when the sharpness of the image farther from the in-focus position of the main subject is not the maximum, the sharpness of the image closer to the in-focus position of the main subject is the maximum, that is, the sharpness is as shown in FIG. Because of the relationship shown in c), the pixel value of the image farther from the focus position of the main subject is output (step S105).

  This process is performed for all pixels. When it is determined that the processing has been completed for all the pixels (step S106), the image composition processing ends.

When the image composition processing is completed, the CPU 24 uses the output image of the image composition unit 38 as the memory control unit 3.
2 is recorded on the recording medium 33 (step S27), and blur-emphasized image shooting is completed.

  As described above, blur enhancement can be performed by selecting a pixel value at which blur is strong in the coordinates of a pixel having low sharpness in an image photographed at the in-focus position B of the main subject. Even when the focal position exceeds three points, selection may be made according to the degree of blur enhancement from a plurality of images at opposite focal positions across the focal position and the focal position where the maximum sharpness is obtained. That is, when it is desired to emphasize blur, it is only necessary to output pixel values of an image with lower sharpness.

  In this embodiment, focus bracket shooting was performed at the in-focus position of the main subject and at A and C separated by d from the in-focus position. However, A and C are not symmetrical with respect to B. The distance from the in-focus position may be different as d and d ′.

  Further, the image composition processing shown in FIG. 10 of the present embodiment may be performed on an image that is not subjected to corresponding point detection and image deformation. In other words, this image composition processing may be performed directly on a plurality of images taken with focus bracketing.

<Third Embodiment>
The blur-enhanced shooting according to the third embodiment of the present invention will be described. FIG. 12 is a block diagram showing an example of the internal configuration of the digital camera 1 according to the third embodiment of the present invention. 1 differs from the block diagram shown in FIG. 1 only in that a filter processing unit 40 is provided instead of the image synthesis unit 38.

  FIG. 13 is a flowchart showing the operation of the blur enhancement shooting mode of the digital camera 1. The parts common to the flowcharts of FIGS. 2 and 8 are denoted by the same reference numerals, and detailed description thereof is omitted. In the blur-enhanced image shooting mode of the digital camera 1 according to the third embodiment of the present invention, focus bracket shooting including shooting at the in-focus position of the main subject and the front and rear focus positions is performed by autofocus, and each captured image is captured. The corresponding points are detected, the captured images are deformed so that the positions of the corresponding points coincide with each other, the sharpness of each pixel is calculated from the deformed image, and the reference image is filtered with a filter coefficient based on the sharpness. By applying this, a blur-emphasized image is obtained.

  As before, when the shutter button of the operation unit 23 is pressed in the blur-enhanced shooting mode, the automatic focus position determination is performed on the main subject (step S81), and then the focal position interval d is determined (step S81). S161).

  In the present embodiment, the focus position set at the time of focus bracket shooting includes the focus position of the main subject and the focus positions before and after the focus position. Here, as shown in FIG. 14, the focus position F3 of the main subject, the focus positions F2 and F4 that are separated from the focus position of the main subject by the focal interval d, and further away from the focus position F3 of the main subject. A total of five points, F1 and F5 at focal positions separated from the focal positions F2 and F4 by a focal distance d, are taken as focal positions for focus bracket photography. Note that in order from the end of the focal position, the number of the image captured at F1 is 1, the number of the image captured at F2 is 2,..., And the number of the image captured at F5 is 5.

FIG. 15 is a graph three-dimensionally showing the relationship between the aperture and focal length at the time of shooting and the distance d between the focal positions at the time of focus bracket shooting. Thus, when the depth of field is shallow, such as when the F value of the diaphragm is small or when the focal length is long, d is small, and conversely, when the F value of the diaphragm is large or the focal length is short, the object is captured. When the depth of field is deep, d is set to be large. The CPU 24 determines the focal distance d based on this setting stored in the ROM area of the main memory 28. As a result, images with different focal points suitable for blur enhancement can be obtained at any aperture and focal length. The focal distance d may not be constant, and may be changed, for example, between the NEAR side and the FAR side when viewed from the focus position of the main subject.

  When the focal distance d is determined from the aperture and focal length at the time of photographing as described above, focus bracket photographing is performed at the focal positions F1 to F5 (steps S21 to S23). The order of shooting may be determined as appropriate. Each captured image is stored in the RAM area of the main memory 28.

  When focus bracket shooting is completed, the corresponding points in each captured image are detected using the image captured at the in-focus position of the main subject as a reference image (step S24). Corresponding points are detected in the same manner as in the first embodiment. When the corresponding point detection process is completed, the image is then deformed (step S25).

  When the image transformation process is completed, the blur amount is calculated (step S162). FIG. 16 is a flowchart showing the operation of the blur amount calculation process. In addition, the same code | symbol is attached | subjected to the part which is common in the flowchart of FIG. 6, and the detailed description is abbreviate | omitted.

  First, the filter processing unit 40 selects a pixel to be processed (step S61). Next, an image is selected (step S62), and the sharpness of the pixel selected in step S61 in the selected image is calculated (step S63). The sharpness is calculated by calculating the absolute value of the output value in the Laplacian filter process, as in the first embodiment. The sharpness of the pixel selected in step S61 is calculated for all images.

  When it is determined that the calculation of the sharpness has been completed for all the images (step S64), the next image number n having the maximum calculated sharpness is calculated (step S191), and the blurring amount k is calculated (step S192). ).

  Here, if the image number taken at the in-focus position of the main subject is c, the blur amount k is expressed by the following equation.

(Equation 1)
k = u × (nc)
Here, u is the degree of blur enhancement specified by the user. If u is larger than 1.0, the blur is emphasized more than the standard setting, and if u is less than 1.0, the blur is reduced more than the standard setting. Image. Note that the blur enhancement degree u can be set by the user operating the operation unit 23. In this embodiment, c = 3.

  This blur amount k is calculated for all pixels. If it is determined that all the pixels have been completed (step S66), the blurring amount calculation process ends.

  When the blur amount calculation process ends, the blur process is performed (step S163). FIG. 17 is a flowchart showing the operation of the blurring process. In this embodiment, a Gaussian filter is used for the blurring process.

  First, the filter processing unit 40 selects a pixel (step S61), and selects a blur amount k of the selected pixel. The blur amount k of each pixel calculated in step S192 in FIG. 16 is stored in the RAM area of the main memory 28, and is read out.

  Next, the absolute value | k | of the read blur amount is compared with a predetermined threshold Th (step S202). If the predetermined threshold Th is larger, this pixel is regarded as a focus area, and a pixel value photographed at the focus position F3 of the subject is output (step S146). If the absolute value | k | of the blurring amount is larger, this pixel is regarded as a region to be blurred and emphasized, and a filter coefficient is determined (step S203).

  As described above, in this embodiment, a Gaussian filter is used for this filter processing. FIG. 18 is a graph showing the relationship between the absolute value | k | of the blur amount and the σ parameter of the Gaussian filter. As shown in FIG. 18, when the absolute value | k | of the blur amount is equal to or greater than a predetermined threshold Th, a σ parameter of a Gaussian filter that is proportional to the absolute value | k | of the blur amount is obtained. A corresponding Gaussian filter coefficient f (x) is determined. The CPU 24 determines the σ parameter based on this relationship stored in the RAM area of the main memory 28.

  FIG. 18B is a graph showing the relationship between the distance x from the target pixel and the filter coefficient f (x) of the Gaussian filter. As shown in FIG. 18B, the Gaussian filter takes a weighted average by increasing the weight of surrounding pixels as the σ parameter increases. Therefore, by increasing the σ parameter in accordance with the magnitude of the absolute value | k | of the blur amount, the degree of smoothing can be increased as the absolute value | k | of the blur amount increases.

  In order to calculate the filter coefficient f (x) from the σ parameter thus obtained, calculation is performed using [Equation 2], and normalization is performed so that the total sum of the calculated filter coefficients becomes 1.

  In the case of a digital filter, f (x) is determined for each discrete position centered on the target pixel. For example, in the case of a 5-tap filter, f (x) = {0.1, 0.2, 0.4, 0.2, 0.1}. In general, normalization is performed so that the sum of the coefficients is 1.0 so that the brightness of the image does not fluctuate. In addition, although it is expressed by a one-dimensional filter coefficient here, two-dimensional filter processing can be performed by sequentially applying this filter in the horizontal direction and the vertical direction.

  As described above, the selected pixel is subjected to the filtering process according to the blur amount, and the output pixel value is calculated (step S145). This process is performed for all pixels, and if it is determined that the process has been completed for all pixels (step S66), the blurring process is terminated. As described above, by performing the filtering process on the reference image captured at the in-focus position B of the main subject, more natural blur enhancement can be performed.

  Note that the filter used for this filter processing is not limited to the Gaussian filter, and other filters may be used as long as they are low-pass filters. For example, a filter having a blur shape corresponding to the characteristics of the diaphragm or lens may be used.

  When the blurring process ends, the CPU 24 records the output image of the filter processing unit 40 on the recording medium 33 via the memory control unit 32, and the omnifocal image shooting ends (step S27).

  In this way, a blur-emphasized image can be obtained.

  Note that the blurring process shown in FIGS. 16 and 17 of the present embodiment may be performed on an image that is not subjected to corresponding point detection and image deformation. In other words, this image composition processing may be performed directly on a plurality of images taken with focus bracketing.

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 11 ... Shooting lens, 12 ... Aperture, 13 ... Ir cut filter, 14 ... Optical low pass filter, 15 ... Solid-state image sensor, 23 ... Operation part, 24 ... CPU, 25 ... Address bus, 26 ... Data bus , 27 ... main memory control section, 28 ... main memory, 32 ... memory control section, 33 ... recording medium, 36 ... corresponding point detection section, 37 ... image deformation section, 38 ... image composition section, 40 ... filter processing section

Claims (6)

Automatic focus position determination means for determining the focus position of the main subject based on the image data;
Focus bracket photographing means for obtaining a plurality of images by moving the focal position discretely with a predetermined amount of movement and sequentially photographing images;
Focus position control means for controlling the moving focus position so as to include the focus position of the main subject determined by the automatic focus position determination means and the focus positions before and after the focus position. A focal position control means for controlling to include a plurality of focal positions before the position and controlling to include a plurality of focal positions after the main subject ;
Corresponding coordinates between a plurality of images photographed by the focus bracket photographing means including an image photographed at a focus position of the main subject and an image photographed at a focal position before and after the focus position of the main subject. Sharpness calculation means for calculating the sharpness of the pixel;
Image combining means for combining the plurality of images;
Recording means for recording the image obtained by the image synthesizing means on a recording medium;
A blur enhancement setting means for setting the blur enhancement;
With
The image synthesizing unit selects, when the sharpness of the pixel photographed at the in-focus position of the main subject among the pixels of the corresponding coordinates between the plurality of images is maximum, and selects the focus of the main subject. When the sharpness of the pixel photographed at the focal position before the position is maximum, the pixel corresponding to the blur enhancement degree is selected from a plurality of images photographed at the focal position after the focal position of the main subject. If the sharpness of the pixel photographed at the focal position after the focusing position of the main subject is the maximum, select from a plurality of images photographed at the focal position before the focusing position of the main subject. An image pickup apparatus comprising: selecting pixels corresponding to the degree of blur enhancement to synthesize an image. A feature point extracting means for extracting a feature point of a subject in an image photographed at an in-focus position of the main subject;
Corresponding point detecting means for detecting corresponding points of the subject corresponding to the feature points in the image taken at the focal position before and after the in-focus position of the main subject;
Image deformation means for deforming an image photographed at a focal position before and after the in-focus position of the main subject so that the coordinates of the corresponding points coincide with the coordinates of the corresponding feature points, respectively.
With
The sharpness calculation means is configured to calculate corresponding coordinates between a plurality of images of an image photographed at a focus position of the main subject and an image photographed at a focal position before and after the focus position of the deformed main subject. The imaging device according to claim 1, wherein the sharpness of the pixel is calculated.   The imaging apparatus according to claim 1, wherein the focal position control unit controls the moving focal position based on at least one of a diaphragm at the time of photographing and a focal length at the time of photographing. An image shot at the focus position of the main subject, a plurality of images shot at a focus position before the focus position of the main subject, and a plurality of images shot at a focus position after the focus position of the main subject Input means for inputting a plurality of images of the same scene photographed at different focal positions including images;
Sharpness calculation means for calculating the sharpness of pixels of corresponding coordinates between the plurality of images;
Image combining means for combining the plurality of images;
Output means for outputting an image obtained by the image composition means;
A blur enhancement setting means for setting the blur enhancement;
With
The image synthesizing unit selects, when the sharpness of the pixel photographed at the in-focus position of the main subject among the pixels of the corresponding coordinates between the plurality of images is maximum, and selects the focus of the main subject. When the sharpness of the pixel photographed at the focal position before the position is maximum, the pixel corresponding to the blur enhancement degree is selected from a plurality of images photographed at the focal position after the focal position of the main subject. If the sharpness of the pixel photographed at the focal position after the focusing position of the main subject is the maximum, select from a plurality of images photographed at the focal position before the focusing position of the main subject. An image processing apparatus, wherein an image is synthesized by selecting pixels according to the degree of blur enhancement . An automatic in-focus position determining step for determining the in-focus position of the main subject based on the image data;
A focus bracket photographing step of obtaining a plurality of images by moving the focal position discretely by a predetermined amount of movement and sequentially photographing images;
A focus position control step of controlling the moving focus position so as to include a focus position of the main subject determined by the automatic focus position determination step and a focus position before and after the focus position of the main subject. A focal position control step for controlling to include a plurality of focal positions before the in-focus position of the main subject and to include a plurality of focal positions after the main subject ;
Corresponding coordinates between a plurality of images photographed by the focus bracket photographing process including an image photographed at a focus position of the main subject and an image photographed at a focal position before and after the focus position of the main subject. A sharpness calculation step for calculating the sharpness of the pixel;
An image synthesis step of synthesizing the plurality of images;
A recording step of recording an image obtained by the image synthesis step on a recording medium;
A blur enhancement setting process for setting the blur enhancement;
With
The image compositing step selects, when the sharpness of a pixel photographed at the in-focus position of the main subject among the pixels of corresponding coordinates between the plurality of images is the maximum, and selects the focus of the main subject When the sharpness of the pixel photographed at the focal position before the position is maximum, the pixel corresponding to the blur enhancement degree is selected from a plurality of images photographed at the focal position after the focal position of the main subject. If the sharpness of the pixel photographed at the focal position after the focusing position of the main subject is the maximum, select from a plurality of images photographed at the focal position before the focusing position of the main subject. An imaging method comprising: selecting pixels according to the degree of blur enhancement and combining images. An image shot at the focus position of the main subject, a plurality of images shot at a focus position before the focus position of the main subject, and a plurality of images shot at a focus position after the focus position of the main subject An input step for inputting a plurality of images of the same scene photographed at different focal positions including images;
A sharpness calculation step of calculating the sharpness of pixels of corresponding coordinates between the plurality of images;
An image synthesis step of synthesizing the plurality of images;
An output step of outputting an image obtained by the image synthesis step;
A blur enhancement setting process for setting the blur enhancement;
With
The image compositing step selects, when the sharpness of a pixel photographed at the in-focus position of the main subject among the pixels of corresponding coordinates between the plurality of images is the maximum, and selects the focus of the main subject When the sharpness of the pixel photographed at the focal position before the position is maximum, the pixel corresponding to the blur enhancement degree is selected from a plurality of images photographed at the focal position after the focal position of the main subject. If the sharpness of the pixel photographed at the focal position after the focusing position of the main subject is the maximum, select from a plurality of images photographed at the focal position before the focusing position of the main subject. An image processing method comprising: selecting pixels according to the degree of blur enhancement to synthesize an image.

Images