JP2020187549A - Image processing device, image processing method, and program - Google Patents

Abstract

To enable emphasizing a main subject by generating a high brightness blurred image when showing in an HDR display.SOLUTION: A high brightness area detection section 204 detects a high brightness area from inputted image data, and a high brightness blurred area detection section 205 uses a detection result of the high brightness area detection section 204, and further detects a high brightness blurred area from the image data. A compensated intensity determination section 206 determines an intensity to be compensated from a feature amount of the high brightness blurred area, and an LPF processing section 207 performs LPF processing to the high brightness blurred area.SELECTED DRAWING: Figure 2

Description

The present invention particularly relates to an image processing apparatus, an image processing method and a program suitable for use in generating an image having a blurred background or the like.

Conventionally, in image processing devices such as digital cameras and digital video cameras, when shooting a person who is the main subject, the background is blurred to lower the contrast and the main subject (person) in focus is emphasized by a shooting method or image processing. Is being done. For example, Patent Document 1 discloses a process of performing a soft focus process on a digital image signal.

Further, conventionally, an SDR (Standard Dynamic Range) display having a maximum display brightness of about 100 nits has been used. However, due to improvements in display technology, HDR (High Dynamic Range) displays capable of displaying up to 1000 to 10000 nits have become available in recent years. As a result, the dynamic range that can be displayed is expanding. For example, Patent Document 2 discloses details about the difference between an SDR signal and an HDR signal, and the brightness that can be displayed by the SDR signal is 0 to 100 nits, but that of the HDR signal is 0 to 10000 nits. A wide dynamic range can be displayed. Therefore, in the conventional SDR display, even if an image having a wide range of luminance values is to be displayed, the maximum display luminance value is limited to about 100 nits, so that the signal on the high luminance side is displayed with the gradation compressed. On the other hand, since the HDR display has a maximum display brightness of up to 10000 nits, it is possible to display an image having a wide range of brightness values as it is.

Japanese Unexamined Patent Publication No. 11-353477 JP-A-2016-213809

However, when shooting a person who is the main subject with a blurred background, if the background has a high brightness value, the background will be blurred while maintaining the high brightness value in the case of an HDR signal, so the background will be blurred. The contrast remains high. When displaying on a conventional SDR display, the gradation of high brightness is compressed, so that the contrast of the blurred image area is lowered and the main subject can be emphasized.

On the other hand, in the case of the HDR display, since a high-brightness image is displayed without compressing the gradation, the contrast does not sufficiently decrease even when the background is blurred. Therefore, the contrast of the blurred image region is kept high, and the high brightness portion of the background becomes more conspicuous than the main subject of the foreground due to this high contrast, and there is a problem that it is difficult to emphasize the main subject of the foreground.

In view of the above-mentioned problems, it is an object of the present invention to generate a high-brightness blurred image so that the main subject can be emphasized when it is displayed on an HDR display.

The image processing apparatus according to the present invention uses a detection means for detecting a high-luminance blur region having a brightness value equal to or higher than a predetermined value and a range of the brightness value within a predetermined range from image data, and the detection means. It is characterized by having a correction means for correcting the high-luminance blur region of the image data based on the detected feature amount of the high-luminance blur region.

According to the present invention, when displaying on an HDR display, a high-brightness blurred image can be generated to emphasize the main subject.

It is a block diagram which shows the internal structure example of the image processing apparatus which concerns on embodiment. It is a block diagram which shows the detailed structure example of the image processing part of 1st Embodiment. It is a flowchart which shows an example of the image processing procedure in 1st Embodiment. It is a figure which shows the example of the image which applies the blur to the background. It is a figure which shows the example of the profile of the luminance value. It is a figure for demonstrating the spread of the histogram of the luminance value. It is a figure for demonstrating the frequency distribution. It is a figure which shows the relationship between the subject distance from a lens, and the distance b from a lens to an image formation position. It is a figure which shows the relationship between the amount of blur and a pixel. It is a block diagram which shows the detailed configuration example of the image processing part of the 2nd Embodiment. It is a flowchart which shows an example of the image processing procedure in 2nd Embodiment. It is a figure which shows the example of the image which applies the blur to the background by the method of 2nd Embodiment. It is a figure which shows the example of the profile of the brightness value after correction. It is a figure for demonstrating the spread of the histogram of the luminance value. It is a figure for demonstrating the characteristic of the luminance value when the contrast is corrected.

(First Embodiment)
Hereinafter, the first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing an example of the internal configuration of the image processing device 101 according to the present embodiment.
In FIG. 1, the optical system 102 includes a lens such as a zoom lens, an optical diaphragm, a focus adjustment function, and the like. The image sensor 103 is a photoelectric conversion element such as a CCD or CMOS, and converts an image of a subject formed by an optical system into an electric signal. The image processing unit 104 performs development processing such as debayer processing on the signal from the image sensor 103, and generates image data after the processing. The recording unit 105 records the image data on a recording medium (not shown). The display unit 106 displays image data, shooting data, and the like. The control unit 107 performs necessary control on each unit of the image processing device 101.

FIG. 2A is a block diagram showing a detailed configuration example in the image processing unit 104 according to the present embodiment.
In FIG. 2A, the image processing unit 104 inputs the image data 202 and generates the image data 203 corrected so as to increase the amount of blur in the high-luminance blur region. The high-luminance region detection unit 204 extracts a region having a high luminance value from the image data 202. The high-luminance blur region detection unit 205 detects the blur region of the image, and together with the detection result of the high-luminance region detection unit 204, detects the high-luminance blur region. The correction intensity determination unit 206 determines the strength of the correction amount for the high-luminance blur region. The low-pass filter (LPF) processing unit 207 performs a low-pass filter processing (hereinafter, LPF processing) on a high-luminance blurred region.

FIG. 3 is a flowchart showing an example of the procedure of image processing by the image processing unit 104 in the present embodiment. Further, FIG. 4 is a diagram showing an example of an image when the image processing of the present embodiment is applied, and FIG. 5 schematically shows a profile of the image when the image processing of the present embodiment is applied. It is a figure. Hereinafter, the operation of the present embodiment will be described with reference to the flowchart of FIG. 3, and the effect of the image processing of the present embodiment will be described with reference to FIGS. 4 and 5.

FIG. 4A is an example of an image when a person who is the main subject is photographed, and is an image in which the person who is the main subject is in the foreground and the buildings are in the background. The image shown in FIG. 4A is an image taken in a state where the entire surface is in focus and the aperture of the lens is narrowed down. In this case, the person in the foreground overlaps the buildings in the background, and the person who is the main subject is not emphasized.

Therefore, in portrait photography, the background is often blurred so that only the person is in focus with the aperture of the lens open. In this case, as shown in FIG. 4B, while the person is in focus, the buildings in the background are blurred, and the person who is the main subject can be emphasized.

Next, the profile of the brightness value when such a person is photographed will be described. FIG. 5 shows the distribution of the luminance values at the positions shown by line 401 in FIG. 4 (c). The horizontal axis in FIG. 5 indicates the position of the image, and the vertical axis indicates the brightness value. Further, the HDR luminance value in FIG. 5 indicates the maximum luminance value that can be displayed on the HDR display, and the SDR luminance value indicates the maximum luminance value that can be displayed on the SDR display.

FIG. 5A is a diagram showing a profile corresponding to the image of FIG. 4A. The central range 501 shows the luminance value of the person who is the subject, and in this example, it shows that the person was photographed to show an intermediate luminance value. In the left and right ranges 502 and 503, the brightness values of the buildings that are the background are shown, and they are brighter than the person, and the maximum brightness value reaches the HDR brightness value. Since the image of FIG. 4A was taken with the entire surface in focus, the profile of the building group is also in a state where the edges are sharp. In particular, in the range 503, there is a dark place (for example, between buildings) in the image, and the image appears as a strong contrast image while maintaining high brightness.

FIG. 5B is a diagram showing a profile corresponding to the image of FIG. 4B. In this case, since the person is in focus, the range 501 has the same profile as in FIG. 5 (a). On the other hand, since the buildings in the background are in a blurred state, the profiles of the buildings in the ranges 504 and 505 are in a wider state than in FIG. 5A. In this case, the maximum luminance value is the HDR luminance value. Further, as shown in the range 505, the dark position of the intermediate image has a contrast lower than that of the range 503 of FIG. 5 (a) due to the blurring effect of the image, but a sufficient brightness difference is maintained.

Here, the difference in display between the HDR display and the SDR display will be described. Since the HDR display can display high brightness, all the brightness values are displayed as they are even if the image has a high brightness portion as shown in the profiles shown in FIGS. 5 (a) and 5 (b). it can. Therefore, it is displayed while maintaining sufficient contrast in a dark place of the intermediate image such as the profile of the range 505.

On the other hand, when displaying on an SDR display, since the maximum display luminance value is limited, it is necessary to compress the gradation on the high luminance side. For example, as shown in FIG. 5C, the range 506 to 508 has a luminance value larger than the SDR luminance value. Therefore, when outputting to the SDR display, a process of compressing the gradation on the high luminance side is performed. FIG. 5D shows a profile when such a compression process is performed. In the range 509 and 510, as a result of compressing the gradation, the maximum luminance value is within the SDR luminance value. Here, in a dark place of the intermediate image in the range 510, the contrast is low due to the compression of the gradation on the high luminance side, and the luminance difference is smaller.

Here, the dark position of the intermediate image is compared between the case where it is displayed on the HDR display and the case where it is displayed on the SDR display. When displayed on an HDR display, as shown in range 505 of FIG. 5B, the difference between the high-luminance portion and the low-luminance portion is large, and a high contrast state is maintained for a blurred image. Due to this high contrast, the person who is the main subject is not emphasized as compared with the SDR display.

Therefore, in the present embodiment, image processing is performed so that the person who is the main subject becomes an emphasized image even when the image is displayed on the HDR display by the following procedure. Hereinafter, the procedure will be described with reference to the flowchart of FIG.

First, in S301, the high-luminance region detection unit 204 detects the high-luminance region from the input image data. In this process, for example, a threshold value Y0 of the brightness value is set, and a region in which the brightness value is equal to or higher than a predetermined value (threshold value Y0 or higher) is detected from the pixels of the image data. For example, when the SDR luminance value is set as the threshold value Y0, the pixels in the range 506 to 508 shown in FIG. 5C are detected. FIG. 4D shows an example of an image in which a high-luminance region is detected. The area shown by the downward slash in FIG. 4D is the high-luminance area.

Next, in S302, the high-luminance blur region detection unit 205 uses the detection result of the high-luminance region detection unit 204 to further detect the high-luminance blur region from the image data.

Here, various methods are possible for detecting the blurred region. For example, there is a method of obtaining a histogram of the luminance value in a small area of an image and observing the spread. That is, when the spread of the histogram is large as shown in FIG. 6A, it is considered that the change in the luminance value is large and the subject is in focus. On the other hand, when the spread of the histogram is small as shown in FIG. 6B, it is considered that the subject is blurred. Alternatively, as another method, the frequency of a small region of the image may be obtained and the blurred region may be detected by the distribution of the frequency. In this case, if there are many high-frequency components as shown in FIG. 7 (a), it is considered that the subject is in focus, and if there are many low-frequency components as shown in FIG. 7 (b), the subject is considered to be in focus. Is considered to be in a blurred state.

On the other hand, the configuration shown in FIG. 2B may be applied instead of the configuration shown in FIG. 2A. In this configuration, when the blurred region is detected, the blurred region can be easily detected by using the shooting information at the time of shooting the image together. That is, in FIG. 2B, the high-luminance blur region detection unit 205 detects the high-luminance blur region using the shooting information 208. Examples of the shooting information at the time of shooting include the focal length of the lens, the aperture, the distance of the subject, and the like. The amount of blur can be calculated from the imaging formula of the lens shown in the following formula (1).
(1 / a) + (1 / b) = (1 / f) ... (1)

Here, a in the equation represents the distance from the lens to the subject, b represents the distance from the lens to the imaging position, and f represents the focal length of the lens. FIG. 8 shows the relationship between the subject distance a from the lens and the distance b from the lens to the imaging position. For example, when the shooting information when the image of FIG. 4B is taken is that the focal length f of the lens is 85 mm and the distance a from the lens to the main subject (person) is 3000 mm, the distance from the lens to the image formation position. b can be calculated as 1 / (1 / 85-1 / 3000) = 87.49 mm.

Further, when the background is sufficiently far away, the distance a from the lens to the subject can be regarded as infinite. In this case, regarding the background, the distance b from the lens to the image formation position is 85 mm, which is 2.49 mm before the image formation position of the main subject (person). Here, if the aperture is F1.8, the amount of background blur is determined to be 2.49 / 1.8 = 1.38 mm. It should be noted that this amount of blur is not necessarily expressed as a numerical value, and may be obtained by the degree of the amount of blur.

In addition, if a distance map in which the distance between the subject and the background is measured by multipoint AF or a defocus map used in fields such as computer vision can be used as shooting information, a more detailed amount of blur can be obtained. ..

Returning to the description of S302 in FIG. 3, in the case of the image of FIG. 4B, the entire background is blurred, so that the blurred region becomes the region of the diagonal line rising to the right in FIG. 4E. The range in which this region and the high-luminance region of FIG. 4D coincide with each other becomes the high-luminance blur region, and becomes the diagonal grid region of FIG. 4 (f). In the profile of FIG. 5 (c), it corresponds to the range 506 to 508.

Next, in S303, the correction intensity determination unit 206 determines the correction intensity from the feature amount of the high-luminance blur region. Here, the correction intensity is the intensity of correction performed as a process for increasing the amount of blur, and in the present embodiment, the process for increasing the amount of blur is performed by the LPF process. In the above calculation, the amount of blur was 1.38 mm, but in the present embodiment, the amount of blur is set to be 1.38 mm or more as the feature amount of the high-luminance blur region. For example, the correction intensity at this time is determined to be doubled, and the amount of blur is determined to be 1.38 × 2 = 2.76 mm.

FIG. 9 shows the relationship between the amount of blur and the pixels. FIG. 9A is an example showing a circle with a blur amount of 1.38 mm on 7 × 7 pixels. In this example, the amount of blur extends to a range of approximately 3 × 3 pixels. On the other hand, FIG. 9B is an example showing a circle with a blur amount of 2.76 mm. In this example, it extends over a range of approximately 5x5 pixels. As the feature amount of the high-luminance blurred region, the luminance value may be used, or the luminance value and the amount of blur may be used in combination.

In the present embodiment, the high-brightness region of the image is obtained, and then the blurred region of the image is obtained to detect the high-brightness blurred region. However, the order is not limited to this, and the blurred region is obtained after the blurred region is obtained. A high-luminance region may be obtained, and these processes may be performed in parallel.

Next, in S304, the LPF processing unit 207 performs LPF processing on the high-luminance blurred region. In the LPF processing, for example, when the amount of blur is 2.76 mm, the average value of the brightness values is obtained for the pixels included in the range of 2.76 mm in diameter centered on the pixel to be processed, and the pixel to be processed is corrected. Let it be the brightness value. For example, the pixel values within the range of the circle shown in FIG. 9B may be averaged. As shown in FIG. 4 (g), the image in which the high-luminance blur region is corrected in this way is an image in which the amount of background blur is increased.

FIG. 5 (e) shows the profile of the image of FIG. 4 (g). As a result of the increased amount of blur, the ranges 511 and 512 in FIG. 5 (e) are wider than the ranges 504 and 505 in FIG. 5 (b). Further, in the dark place of the intermediate image in the range 510, the luminance difference from the maximum luminance value is smaller and the contrast is low. Therefore, even if the background is blurred, the person who is the main subject can be emphasized.

When displaying an image on the SDR display, the image processing unit 104 performs processing for SDR. In this case, it is not necessary to perform the correction process or the like according to the procedure shown in the present embodiment.

As described above, according to the present embodiment, the high-luminance blurred region is detected, the amount of blur is increased, and the LPF processing is performed. As a result, even when the image is displayed on the HDR display, the contrast of the high-brightness blurred image can be lowered, and an image in which the main subject is emphasized can be obtained.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. In the present embodiment, an example of detecting the contrast of an image and correcting a blurred image so as to reduce the contrast will be described. Since the configuration of the entire image processing apparatus of this embodiment is the same as that of FIG. 1, the description thereof will be omitted.

FIG. 10 is a block diagram showing a detailed configuration example of the image processing unit 104 in the present embodiment. The description of the portion overlapping with the configuration shown in FIG. 2 will be omitted.
In FIG. 10, the contrast detection unit 1003 detects the contrast from the high-luminance blur region and the peripheral region thereof. The correction intensity determination unit 1004 determines the strength of the correction amount for the high-luminance blur region. The contrast correction unit 1005 corrects so as to reduce the contrast in the high-luminance blur region.

FIG. 11 is a flowchart showing an example of the procedure of image processing by the image processing unit 104 in the present embodiment. Since the processes of S1101 and S1102 of FIG. 11 are the same as those of S301 and S302 of FIG. 3, the description thereof will be omitted.

In the next S1103, the contrast detection unit 1003 detects the contrast from the image data. In this process, for example, a histogram of the luminance value is generated in the high-luminance blurred region and the region including the surrounding pixels, and the contrast is detected from the spread. Here, the range of peripheral pixels may be set according to the amount of blur. For example, when a high-luminance blurred region as shown in FIG. 4 (f) is detected, a histogram of the luminance value is generated for the region filled in the grid pattern of FIG. 12 (a). Examples of the histogram in this case are shown in FIGS. 14 (a) and 14 (b).

FIG. 14 (a) is a diagram showing an example of a histogram of luminance values in the grid-like region on the left side of FIG. 12 (a). Since this region includes dark regions between buildings, the contrast is high in the histogram range 1401. On the other hand, FIG. 14B is a diagram showing an example of a histogram of the luminance values in the grid-like region on the right side of FIG. 12A. In the histogram range 1402, the contrast is low. That is, the grid-like region on the left side of FIG. 12A is a high-brightness blurred region, and the contrast is high including the peripheral pixels, and the person who is the main subject cannot be emphasized unless correction is performed. ..

Therefore, in the next S1104, the correction intensity determination unit 1004 determines the contrast correction amount according to the high or low contrast as the feature amount of the high-luminance blur region. In the present embodiment, the amount of contrast correction is determined according to the spread of the histogram. The brightness value and the amount of blur described in the first embodiment may be used in combination as the feature amount of the high-brightness blurred region.

Then, in S1105, the contrast correction unit 1005 corrects the contrast based on the correction amount determined in S1104. In the case of the example shown in FIG. 12A, the contrast is corrected in the grid-like region on the left side. In the contrast correction, as shown in FIG. 15, in a range of a certain value or more of the input luminance value, the low-luminance side is raised and the high-luminance side is lowered to lower the contrast. As described above, in the present embodiment, the correction for substantially increasing the amount of blurring is performed by lowering the contrast. FIG. 15A shows the characteristics of the luminance value when the contrast correction amount is small in the range 1501, and FIG. 15B shows the characteristics of the luminance value when the contrast correction amount is large in the range 1502. Shown.

FIG. 14C shows a histogram of the luminance values after the contrast is corrected. Further, FIG. 12B shows an image after contrast correction, and FIG. 13 shows a profile of the luminance value. As shown in FIG. 13, the contrast becomes small at the position 1301 corresponding to the grid-like region on the left side of FIG. 12A, and the person who is the main subject can be corrected so as to be emphasized. Note that the correction is not performed at the position 1302 corresponding to the grid-like region on the right side of FIG. 12 (b).

As described above, according to the present embodiment, the high-luminance blurred region is detected and the contrast is corrected. As a result, even when the image is displayed on the HDR display, the contrast of the blurred area can be lowered and the main subject can be emphasized.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

204 High-brightness area detection unit 205 High-brightness blur area detection unit 206 Correction intensity determination unit 207 LPF processing unit

Claims (10)

A detection means for detecting a high-luminance blurred region having a luminance value equal to or higher than a predetermined value and having a luminance value range within a predetermined range from image data.
A correction means that corrects the high-luminance blur region of the image data based on the feature amount of the high-luminance blur region detected by the detection means.
An image processing device characterized by having. The image processing apparatus according to claim 1, wherein the feature amount includes a brightness value in the high-luminance blur region. The image processing apparatus according to claim 1 or 2, wherein the correction means corrects the amount of blur in the high-luminance blur region by low-pass filter processing. The image processing apparatus according to claim 1, wherein the feature amount includes contrast in the high-luminance bokeh region. The image processing apparatus according to claim 1 or 4, wherein the correction means performs correction for lowering the contrast in the high-luminance blur region. The image processing apparatus according to any one of claims 1 to 5, wherein the detection means detects the high-luminance blurred region using shooting information at the time of shooting. The image processing apparatus according to claim 6, wherein the shooting information includes at least one of an aperture value at the time of shooting, a focal length of a lens, a distance of a subject, a distance map, and a defocus map. The image processing apparatus according to any one of claims 1 to 7, wherein the predetermined value is a maximum luminance value when displayed on an SDR display. A detection step of detecting a high-luminance blurred region having a luminance value equal to or higher than a predetermined value and having a luminance value range within a predetermined range from image data.
A correction step of correcting the high-luminance blur region of the image data based on the feature amount of the high-luminance blur region detected in the detection step,
An image processing method characterized by having. A detection step of detecting a high-luminance blurred region having a luminance value equal to or higher than a predetermined value and having a luminance value range within a predetermined range from image data.
A correction step of correcting the high-luminance blur region of the image data based on the feature amount of the high-luminance blur region detected in the detection step,
A program that lets your computer run.