JP2011211523A - Image processing method, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method of creating a blur-controlled image with many blur levels even when a small number of images are used.SOLUTION: In the image processing method, sharpness related to corresponding point pixels are determined for each image from n images captured by changing a focusing position, the corresponding point pixels are grouped based on relation of n sharpness levels, and the corresponding point pixels belonging to each group are blurred at an emphasis level specific to each group. In the image processing method, regarding two images captured at different focusing positions, in addition to a sharpness region jp=0 where the corresponding point pixels in one image are mainly included and a sharpness region jp=2 where the corresponding point pixels in the other image are mainly included, an intermediate sharpness region jp=1 is set, and the regions are defined as a group. Based on the relation of the sharpness levels of the corresponding point pixels of the two images, the corresponding point pixels are included in any of the sharpness regions jp=0, jp=1, jp=2 for grouping.

Description

  The present invention relates to an image processing method, and more particularly to an image processing method for creating a blur enhanced image and a blur control image.

  The present invention also relates to a photographing apparatus that performs the image processing method as described above.

  Conventionally, as shown in, for example, Patent Documents 1 and 2, in an image such as a photograph, a blur-enhanced image in which blur in a certain area is emphasized, or blur control in which the blur amount in several areas is changed for each area. Techniques for creating images are known. According to such a technique, for example, the main subject is not blurred, but the background and foreground portions are given a large amount of blur to create a blur control image that gives the impression that the main subject is vividly raised. Can do.

  The blur-enhanced image and the blur control image described above are created using a plurality of images obtained by so-called focus bracket shooting, in which the same subject is shot while changing the focus position, as described in Patent Document 1. Often. According to this focus bracket shooting, you can obtain an image that is focused and has high sharpness in a certain part of the image, and an image that is not focused and low in sharpness in that part. It is possible to create a blurred image using a portion where the sharpness is low.

JP 2008-271240 A JP 2002-84444 A

  In the blur-enhanced image and the blur control image described above, if the number of stages of blurring is small, an unnatural blurring boundary may occur, or blurring unevenness may occur due to the influence of noise on the flat part of the image. is there.

  In order to prevent such a problem, it is necessary to secure a large number of stages of blurring. In multiple images obtained by focus bracket shooting, the number of sharpness differences in each part of the image is essentially the same as the number of shots, so it is inevitably necessary to secure a large number of blur amounts. It is necessary to take a large number of images and subject them to image processing. However, when a large number of images are taken and these many images are subjected to image processing, it takes a long time to create a blur control image.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing method that can secure a larger number of stages of blurring amounts by using a small number of images.

  Another object of the present invention is to provide a photographing apparatus that performs the above-described image processing method.

An image processing method according to the present invention includes:
From each of n (2 ≦ n) images obtained by photographing the same subject with different focus positions, the sharpness of corresponding point pixels whose in-image positions match is determined for each image,
Based on the n kinds of sharpness magnitude relationships, the corresponding pixel is grouped into one of a plurality of predetermined groups,
In the image processing method for performing the blurring process with the emphasis degree specific to each group on the corresponding point pixels belonging to each group,
For two images taken at different in-focus positions, the sharpness region where the corresponding pixel in one image is mainly included and the corresponding pixel in the other image are mainly included In addition to the sharpness region, the intermediate sharpness region is set, and those regions are grouped,
Based on the magnitude relationship of the sharpness of corresponding point pixels of the two images, the corresponding point pixels belong to any one of the plurality of sharpness regions, and the grouping is performed.

  In the image processing method of the present invention, it is not necessary to group all the corresponding point pixels based on the magnitude relationship of the sharpness, and some corresponding point pixels may be grouped according to other criteria. Good. That is, for example, based on the sharpness of corresponding point pixels of n images, flat portions in those images are detected, and corresponding point pixels included in the detected flat portions are included in the intermediate sharpness region. You may make it belong and perform grouping.

  In the image processing method of the present invention, the intermediate sharpness region is set to be wider as the representative luminance value of the image captured by focusing on the main subject among the n images is lower. It is desirable.

  In the image processing method of the present invention, it is desirable that the intermediate sharpness region is set wider as the shooting sensitivity when shooting the n images is higher.

  Furthermore, in the image processing method of the present invention, flat portions in the images are detected based on the sharpness of the corresponding point pixels of the n images, and blur processing is performed on the detected flat portions. It is desirable to perform a smoothing process before performing.

The present invention also provides a photographing apparatus capable of performing the above-described image processing method.
In a photographing apparatus having a function of photographing n identical subjects (2 ≦ n) by changing the focus position to each other,
From each of the n images, the sharpness for the corresponding point pixel with the same position in the image is obtained for each image,
Based on the n kinds of sharpness magnitude relationships, the corresponding pixel is grouped into one of a plurality of predetermined groups,
For the corresponding point pixel belonging to each group, it has an image processing function for performing a blurring process with an emphasis degree specific to each group,
For two images taken at two in-focus positions adjacent to each other, a sharpness region that mainly includes corresponding point pixels in one image and a corresponding point pixel in the other image are mainly included. In addition to the sharpness regions that become, set intermediate sharpness regions between them, and make those regions into the group,
Based on the relationship between the sharpness levels of corresponding point pixels of the two images, the corresponding point pixels belong to one of the plurality of sharpness regions and have the function of performing the grouping. It is.

  In the photographing apparatus according to the present invention, when a predetermined number of images of the same subject are photographed by changing the focusing position, if the number of focusing points does not reach the number, the intermediate position between the focusing points is reached. It is desirable to have a shooting function.

According to the image processing method of the present invention,
From each of n (2 ≦ n) images obtained by photographing the same subject with different focus positions, the sharpness of corresponding point pixels whose in-image positions match is determined for each image,
Based on the n kinds of sharpness magnitude relationships, the corresponding pixel is grouped into one of a plurality of predetermined groups,
When performing the blurring process on the corresponding point pixels belonging to each group with the emphasis level specific to each group,
For two images taken at different in-focus positions, the sharpness region where the corresponding pixel in one image is mainly included and the corresponding pixel in the other image are mainly included In addition to the sharpness region, the intermediate sharpness region is set, and those regions are grouped,
Based on the magnitude relationship between the sharpness of corresponding point pixels of the two images, the corresponding point pixels belong to one of the plurality of sharpness regions and the grouping is performed. Since the sharpness region is set, it is possible to set the number of blur amount steps larger than the number of images.

  If a large number of stages of blurring can be secured in this way, an unnatural blurring boundary may occur due to the small number of stages of blurring, and blurring unevenness may occur due to the influence of noise in the flat part of the image. It becomes possible to prevent.

  In the image processing method of the present invention, in particular, the intermediate sharpness region is set to be wider as the representative luminance value of an image captured by focusing on the main subject among n images is lower. In the case of the above, the following effects can be obtained. In other words, when the brightness value is generally low, noise is likely to occur in the image, and blurring unevenness is likely to occur due to the influence of noise in the flat portion of the image, but by setting a wider intermediate sharpness region Since erroneous determination of grouping due to the influence of noise is suppressed, this blur unevenness can be reduced.

  In the image processing method of the present invention, in particular, the higher the shooting sensitivity when shooting n images, the more the blurring unevenness can be reduced even when the intermediate sharpness region is set wider. . That is, an input image shot with high sensitivity is more likely to generate noise than an input image shot with low sensitivity. Therefore, if the input image is taken with high sensitivity, the noise level can be reduced by setting the intermediate sharpness region wider, as in the case where the brightness value is generally low. Since the erroneous determination of the grouping due to the influence of the above is suppressed, the blurring unevenness can be reduced.

Further, in the image processing method of the present invention, in particular, based on the sharpness of corresponding point pixels of n images, flat portions in those images are detected, and the blur processing is performed on the detected flat portions. When the smoothing process is performed before performing the following, the following effects can also be obtained. In other words, noise is likely to occur in the flat part of the image, but if the smoothing process is performed on the flat part in this way before the blurring process, the noise is reduced. Can be removed.

  On the other hand, according to the photographing apparatus of the present invention having the configuration as described above, the image processing method of the present invention can be carried out.

  In particular, when the photographing apparatus of the present invention takes a predetermined number of images of the same subject by changing the focusing positions, if the number of focusing points does not reach the number, the intermediate position of the focusing points If it has the function to shoot with, a large number of shots can be secured even if there are few focus points. In this way, if the number of shots is increased, the number of steps of the blur amount can be secured, so the number of steps of the blur amount is so small that an unnatural blur boundary occurs or blur unevenness occurs due to the influence of noise in the flat part of the image. Can be prevented.

1 is a schematic configuration diagram showing a photographing apparatus that performs an image processing method according to an embodiment of the present invention. The flowchart which shows the flow of the image processing performed in the apparatus of FIG. The flowchart which shows a part of process shown in FIG. 2 in detail FIG. 3 is a conceptual diagram illustrating a pixel sorting area applied to the process shown in FIG. FIG. 3 is a conceptual diagram illustrating a pixel sorting area applied to the process shown in FIG. FIG. 3 is a conceptual diagram illustrating a pixel sorting area applied to the process shown in FIG. The perspective view which shows the microchannel type | mold sensor chip used for the said measuring apparatus. The figure explaining the state at the time of the biological substance detection in the sensor chip of FIG. The graph which shows the example of a change characteristic of the internal pressure of a communicating path in the measuring apparatus of FIG. FIG. 1 is a graph showing an example of measurement calculation values in the measurement apparatus of FIG. FIG. 1 is a graph showing an example of measurement calculation values in the measurement apparatus of FIG. The flowchart which shows the flow of another image processing which can be performed in the apparatus of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a photographing apparatus 1 that performs an image processing method according to an embodiment of the present invention. The photographing apparatus 1 is formed as a digital camera, for example, and takes a photographing lens system 10 including a zoom lens 11, a focus lens 12, a diaphragm 13, and the like, and an image of a subject connected by the photographing lens system 10. An image sensor 14 comprising a CCD or the like, a signal processing means 15 for processing an output signal of the image sensor 14, an image processing means 16 for image processing an image signal output from the signal processing means 15, and the image processing means. 16 has a monitor 17 for displaying a photographed subject image based on an image signal output from 16 and a control means 18 for controlling the operation of each of the elements 11-17.

  The image processing means 16 is provided for displaying on the monitor 17 an image on which blur control has been performed or an image on which other processing has been performed. When performing blur processing, the image processing means 16 The process shown in FIG. 2 is performed. Hereinafter, this process will be described.

  When the blurring process is started, next, in step S1, a process of calculating the sharpness for each pixel of a plurality of images is performed. As the plurality of images, two or three images, which are so-called focus bracket photographing, which is photographed while changing the focus position by operating the focus lens 12 shown in FIG.

  Next, in step S2, based on the calculated sharpness, it is determined whether or not there is a flat portion, that is, an area in which the sharpness change is extremely low. If it is determined that there is a flat part, then in step S3, a process of smoothing the flat part of the reference image (image focused on the main subject) of those images is performed. Processing for setting the blurring amount in the blurring processing to be performed to a predetermined fixed value is performed.

  On the other hand, if it is determined in step S2 that there is no flat portion, then in step S5, grouping processing by sharp surface calculation is performed. This process will be described in detail later. Then, in step S6, a group-specific blur amount (enhancement degree) for the grouped pixels is calculated as described later. Next, in step S7, the pixels of each group are subjected to blurring processing using the calculated group-specific blurring amount.

  Here, depending on the setting of the inherent blur amount, for example, the main subject that is at an intermediate distance from the camera is not blurred, and the foreground that is closer to the main subject, that is, on the camera side, On the contrary, the background located farther than the main subject is greatly blurred, and an image or the like in which only the main subject is clearly projected can be formed. This technique is conventionally known as so-called blur control. At this time, the blur processing is performed on the pixels in the flat portion with the fixed blur amount.

  Note that noise is likely to occur in the flat part of the image, but noise is reduced by performing the smoothing process on the flat part before the blurring process in this way. Blur unevenness can be removed.

  In this embodiment, it is determined in step S2 whether or not there is a flat portion in the image. If it is determined that there is a flat portion, the processing in steps S3 and S4 is performed next. As shown in FIG. 12, the processing of these steps S2 to S4 may not be performed, and the grouping processing from step S1 to step S5 may be performed regardless of the presence or absence of the flat portion.

  Next, the processing performed in step S5 shown in FIG. 2 will be described with reference to FIG. 3 showing the detailed processing flow.

  In the present embodiment, it is assumed that the image processing unit 16 in FIG. 1 is input with image signals for two or three images obtained by photographing the same subject with different focus positions. In order to perform grouping, the image processing means 16 stores a sorting area conceptually shown in FIGS. Here, sa [n] shown in FIGS. 3 to 6 indicates the sharpness of each pixel in the image of the image number n. It should be noted that the smaller the number of n, the nearer the image, that is, the image taken while focusing on the side closer to the photographing apparatus 1.

  Here, when the corresponding point pixel sharpness sa [0] is m and the sharpness sa [1] is n, when they are plotted in FIG. 4, a point Q in the figure is obtained.

  The sorting area shown in FIG. 4 is used when two images are input to the image processing means 16, and the area is divided into three by two straight lines having slopes R1 and R2. The lower side of the former straight line is the Near sharp region, the upper side of the latter straight line is the Far sharp region, and the region between them is the virtual surface region. Here, these areas are defined in the order of jp = 0, jp = 1, and jp = 2 from the Near side.

  The sorting area shown in FIGS. 5 and 6 is used when three images are input to the image processing means 16. More specifically, in FIG. 5, the corresponding point pixels (pixels whose positions in the image match) in the three images are sharper toward the Near side, that is, an image that is photographed by focusing on the side closer to the photographing apparatus 1. 6 is used when the corresponding point pixel in the three images is sharper as far as the Far side, that is, the image taken with focusing on the farther side from the photographing apparatus 1. Used when the sharpness is higher.

  The sorting area shown in FIG. 5 and FIG. 6 is divided into five areas with four straight lines having an inclination of R3, R4, R5, and R6, and the lower side of the straight line of the inclination R3 is a Near sharp area, The area between the straight line with the slope R4 and the straight line with the slope R5 (the vertical axis in FIG. 5 is the horizontal axis in FIG. 6) is the Center sharp area, and the upper side of the straight line with the slope R6 is the Far sharp area. The area between the Near sharp area and the Center sharp area is a virtual surface area, and the area between the Center sharp area and the Far sharp area is also a virtual surface area. These areas are defined in order from the Near side by the numbers jp = 0, jp = 1, jp = 2, jp = 3, and jp = 4.

  When the processing of FIG. 3 is started, first, in step S11, it is determined whether or not the number of input images N is two. If it is determined that there are two images, then in step S12, the ratio of the corresponding point pixel sharpness sa [1] to the corresponding point pixel sharpness sa [0], that is, the value of (sa [1] / sa [0]) The slope R1 is compared. As a result of this comparison, when (sa [1] / sa [0]) ≦ R1, the corresponding pixel is classified as belonging to the Near sharp region with jp = 0 (step S13).

  On the other hand, if (sa [1] / sa [0]) ≦ R1, the value of (sa [1] / sa [0]) and the slope R2 are compared in step S14. Here, when (sa [1] / sa [0]) ≧ R2, the corresponding point pixel is classified as belonging to the Far sharp region of jp = 2 (step S15). On the other hand, if (sa [1] / sa [0]) ≧ R2 is not satisfied, the corresponding point pixel is classified as belonging to the virtual plane region of jp = 1 (step S16).

  The above processing is performed for all the corresponding point pixels, and each corresponding point pixel is classified as belonging to one of the three regions. In order to determine corresponding point pixels between two images, one or both of the images are enlarged or contracted, and the pixels are associated with each other based on pixel values or the like in both images. However, they may be performed by a known method.

  With respect to the corresponding point pixels divided into the three groups as described above, a specific blur amount is calculated for each group in step S6 of FIG. Next, in step S7, the corresponding point pixels of each group are subjected to blurring processing using the calculated group-specific blurring amount. A conventionally known process can be applied to the blurring process performed in step S7. Since the present invention is not directly related to the blurring process itself, detailed description thereof is omitted.

  As described above, in this example, the Near sharp region (jp = 0) in which the corresponding point pixels in the Near image are mainly included, and the Far sharp region in which the corresponding point pixels in the Far image are mainly included. In addition to (jp = 2), an intermediate virtual plane area (jp = 1) between them is set, and corresponding point pixels are grouped by them, so that the same focus position is changed. While only two images of the subject are used, the corresponding point pixels can be sorted into three groups. If the number of steps of the blur amount (enhancement level) can be increased to three in this way, the number of steps of the blur amount is so small that an unnatural blur region occurs in the blur control image or noise is generated in the flat portion of the image. It is possible to prevent blurring unevenness from occurring due to the influence.

  Note that it is possible to increase the number of stages of blurring to four or more by providing two or more virtual surface areas between the Near sharp area (jp = 0) and the Far sharp area (jp = 2). .

  In the processing of FIG. 2 described above, if it is determined in step S2 that the image has a flat portion, the corresponding point pixel included in the flat portion is automatically set to the intermediate point regardless of the sharpness. The virtual surface area (jp = 1) may be sorted.

  Next, returning to FIG. 3, a case where three images are used will be described. In this case, since it is determined in step S11 that the number of input images N is not two, that is, three, in step S17, the corresponding point pixel sharpness sa [0] and the corresponding point pixel sharpness sa [2] is compared. If it is determined that sa [0] <sa [2], that is, if the corresponding point pixel is determined to be sharper toward the Near side, the process of step S18 is performed next. If not, the process of step S23 is performed. Processing is done.

  In steps S18 to S22, the same processing as in steps S12 to S16 described above is performed with the comparison targets as slopes R3 and R4. In steps S23 to S27, the same processing as in steps S12 to S16 described above is performed. The processing is performed with the comparison target as the corresponding point pixel sharpness sa [1], sa [2] and the gradients R5, R6. Thereby, each corresponding point pixel is classified into one of the five areas indicated by numbers jp = 0, jp = 1, jp = 2, jp = 3, and jp = 4 shown in FIGS. Become.

  For the five groups as described above, a unique blur amount is calculated for each group in step S6 of FIG. Next, in step S7, the corresponding point pixels of each group are subjected to blurring processing using the calculated group-specific blurring amount.

  As described above, in this case, the Near sharp region (jp = 0) in which the corresponding point pixel is mainly included in the Near image and the Far sharp region in which the corresponding point pixel is mainly included in the Far image. In addition to (jp = 4) and the Center sharp region (jp = 2) between Near image and Far image, two intermediate virtual surface regions (jp = 1 and jp = 3) Since the corresponding point pixels are set and grouped by them, the corresponding point pixels can be classified into five groups while using only three images obtained by photographing the same subject with different focus positions. ing. If the number of steps of the blur amount can be increased to five in this way, the number of steps of the blur amount is so small that an unnatural blur region is generated in the blur control image, or it is affected by noise in the flat portion of the image. It is possible to prevent blurring unevenness from occurring.

  Note that only one of the two virtual surface areas (jp = 1 and jp = 3) may be provided, and the number of stages of blurring may be set to four. In this case, the three images are used as they are. The effect described above can be obtained in the same manner as compared to the case where the number of stages of blurring is three.

  In addition, it is desirable that the size of the virtual surface area described above is appropriately changed according to the condition of the input image. For example, considering the virtual surface area (jp = 1) in FIG. 4 in the case of two-image input, when the luminance value (for example, the average value of all pixels) of the input image is generally high, as shown in FIG. If the luminance value is relatively low, it is desirable to make it relatively wide as shown in FIG. That is, when the brightness value is generally low, noise is likely to occur in the image, and blurring unevenness is likely to occur due to the influence of noise in the flat part of the image, but this blurring can be achieved by setting the virtual surface area wider. Unevenness can be reduced.

  In addition, an input image captured with high sensitivity is more likely to generate noise than an input image captured with low sensitivity. Therefore, when the input image is taken with low sensitivity, the virtual plane area is relatively narrow as shown in FIG. 7, and when the input image is taken with high sensitivity, FIG. As shown in FIG. By doing so, also in this case, it is possible to reduce blurring unevenness that easily occurs due to the influence of noise.

  In the photographing apparatus 1 such as the camera shown in FIG. 1, when photographing a predetermined number of the same subject with different focus positions, the apparatus can photograph three images, for example, but the situation of the subject Depending on the number, there may be no appropriate focusing point for that number. 9 to 11 illustrate this by the relationship between the position of the focus lens 12 as shown in FIG. 1 and the focus evaluation value detected during AF (automatic focusing) processing using the focus lens 12. It is shown.

  FIG. 9 shows that, for example, the AF mechanism of the photographing apparatus 1 gives a high focus evaluation value to all of the main subject, the background, and the foreground, and these can all be photographed at an appropriate focus point (focus position). It shows the case. On the other hand, in the example of FIG. 10, the focus point of the main subject is on the background side, so there is no background focus point. In the example of FIG. 11, since the focus point of the main subject is on the foreground side, there is no foreground focus point.

  Therefore, in the situation shown in FIG. 10, the photographing apparatus 1 shown in FIG. 1 has an intermediate photographing position between the two in-focus points (the foreground in-focus point and the main subject are in alignment) even if the in-focus evaluation value is low. It is desirable that the camera be configured to shoot between the focal points. Alternatively, in the situation shown in FIG. 11, the photographing apparatus 1 may have an intermediate photographing position between the two in-focus points (between the in-focus point of the background and the in-focus point of the main subject) even if the in-focus evaluation value is low. ) Is desirable to shoot.

  In this way, it is possible to avoid the fact that the number of images input to the image processing means is reduced because there are few in-focus points, and consequently the number of stages of blurring amount is reduced.

DESCRIPTION OF SYMBOLS 1 Shooting device 10 Shooting lens system 11 Zoom lens 12 Focus lens 14 Image sensor 15 Signal processing means 16 Image processing means 18 Control means

Claims (7)

From each of n (2 ≦ n) images obtained by photographing the same subject with different focus positions, the sharpness of corresponding point pixels whose in-image positions match is determined for each image,
Based on the n kinds of sharpness magnitude relationships, the corresponding pixel is grouped into one of a plurality of predetermined groups,
In the image processing method for performing the blurring process with the emphasis degree specific to each group on the corresponding point pixels belonging to each group,
For two images taken at different in-focus positions, the sharpness region where the corresponding pixel in one image is mainly included and the corresponding pixel in the other image are mainly included In addition to the sharpness region, the intermediate sharpness region is set, and those regions are grouped,
An image processing method, wherein the grouping is performed by assigning the corresponding point pixel to one of the plurality of sharpness regions based on a magnitude relationship between the sharpness levels of corresponding point pixels of the two images. Based on the sharpness of the corresponding point pixels of the n images, detecting a flat portion in those images,
The image processing method according to claim 1, wherein the corresponding point pixels included in the detected flat portion belong to the intermediate sharpness region to perform the grouping.   3. The intermediate sharpness region is set wider as the representative luminance value of an image photographed by focusing on a main subject among the n images is lower. Image processing method.   The image processing method according to any one of claims 1 to 3, wherein the intermediate sharpness region is set wider as the photographing sensitivity when the n images are taken is higher. Based on the sharpness of the corresponding point pixels of the n images, detecting a flat portion in those images,
5. The image processing method according to claim 1, wherein smoothing processing is performed on the detected flat portion before performing the blurring processing. In a photographing apparatus having a function of photographing n identical subjects (2 ≦ n) by changing the focus position to each other,
From each of the n images, the sharpness for the corresponding point pixel with the same position in the image is obtained for each image,
Based on the n kinds of sharpness magnitude relationships, the corresponding pixel is grouped into one of a plurality of predetermined groups,
For the corresponding point pixel belonging to each group, it has an image processing function for performing a blurring process with an emphasis degree specific to each group,
For two images taken at two in-focus positions adjacent to each other, a sharpness region that mainly includes corresponding point pixels in one image and a corresponding point pixel in the other image are mainly included. In addition to the sharpness regions that become, set intermediate sharpness regions between them, and make those regions into the group,
Based on the magnitude relationship of the sharpness of corresponding point pixels of the two images, the photographing has a function of performing the grouping by assigning the corresponding point pixel to one of the plurality of sharpness regions apparatus.   When shooting the same number of subjects with different focus positions, if the number of in-focus points has not reached that number, it has a function to shoot at an intermediate position between those in-focus points The imaging device according to claim 6.

Images