JP5024632B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly to an image processing apparatus, method, and program capable of accurately obtaining a depth value of a subject in an image and processing the image based on the obtained depth value. About.

  There has been proposed a technique for setting a depth value for each subject in an image and processing the image according to the depth value (Patent Document 1). Further, a process for extracting the depth value of the subject from the image and performing the above-described image processing in accordance with the extracted depth value has been proposed.

JP 2002-197486 A

  However, when the depth value is detected from the contour component or the high frequency component of the image, the depth value is often based only on the luminance information, and therefore the depth and the high frequency component are very small in the flat portion. May not be obtained accurately, and image processing may not be performed properly. Even in a dark region, the contour and the high-frequency component are small, so that an accurate depth value may not be obtained, and there is a possibility that image processing cannot be performed properly.

  The present invention has been made in view of such a situation. In particular, the depth value can be accurately set even in a flat portion such as a dark portion in an image.

  An image processing apparatus according to an aspect of the present invention integrates focus information extraction means for extracting high-frequency components as focus information from luminance signals of peripheral pixels for each pixel of an input image, and focus information of peripheral pixels for each pixel. The region integration means for generating the integrated focus information and replacing the input for each pixel with a value obtained based on focus information larger than its own focus information among the focus information of surrounding pixels. Dark part processing means for generating dark part focus information of an image, illumination component extraction means for extracting an illumination component by integrating the luminance signal of each pixel with the luminance signal of surrounding pixels, the integrated focus information, and the dark part focus A light / dark integration unit that calculates a depth value for each pixel by a product-sum operation with a coefficient using the illumination component, a normalization unit that normalizes the depth value for each pixel, and the normalization unit Tone curve control means for controlling the normalized depth value by a tone curve, saturation detection means for detecting the saturation for each pixel, and the depth value controlled by the tone curve by the tone curve control means And a saturation control means for controlling the brightness according to the saturation.

  Perspective control means for controlling luminance and color difference signals for each pixel of the input image by a coefficient based on the depth value controlled by the tone curve by the saturation control means may be further included. it can.

  A sharpness control means for controlling the brightness sharpness, a contrast control means for controlling the brightness contrast, and a color difference control means for controlling a color difference signal for each pixel may be further included. The sensation control means uses the brightness controlled by the sharpness control means and the brightness controlled by the contrast control means in addition to the brightness for each pixel to control the brightness by the coefficient, and the color difference signal The color difference signal is controlled by a coefficient based on the depth value using the color difference signal controlled by the color difference control means.

  An image processing method according to one aspect of the present invention integrates focus information of a peripheral pixel for each pixel, and a focus information extraction step for extracting a high-frequency component as focus information from a luminance signal of the peripheral pixel for each pixel of an input image. Region integration step for generating the integrated focus information, and for each pixel, by replacing the focus information of peripheral pixels with a value obtained based on focus information larger than its own focus information, the input A dark portion processing step for generating dark portion focus information of an image, an illumination component extraction step for extracting an illumination component by integrating the luminance signal of each pixel with a luminance signal of surrounding pixels, the integrated focus information, and the dark portion focus And a light and dark integration step of calculating a depth value for each pixel by a product-sum operation with a coefficient using the illumination component, and a normalization step for normalizing the depth value for each pixel. And a tone curve control step for controlling the depth value normalized by the normalization step processing by a tone curve, a saturation detection step for detecting saturation for each pixel, and the tone curve control step processing. And a saturation control step of controlling the depth value controlled by the tone curve based on the saturation.

  A program according to one aspect of the present invention integrates focus information of a peripheral pixel for each pixel, and a focus information extraction step that extracts a high-frequency component from the luminance signal of the peripheral pixel as focus information for each pixel of the input image. In the region integration step for generating the integrated focus information, and for each pixel, by replacing the focus information of peripheral pixels with a value obtained based on focus information larger than its own focus information, the input image A dark portion processing step for generating dark portion focus information, an illumination component extraction step for extracting an illumination component by integrating the luminance signal of each pixel with a luminance signal of surrounding pixels, the integrated focus information, and the dark portion focus information, And a light-dark integration step for calculating a depth value for each pixel by a product-sum operation using a coefficient using the illumination component, and a normalization step for normalizing the depth value for each pixel. The tone curve control step for controlling the depth value normalized by the normalization step processing with a tone curve, the saturation detection step for detecting the saturation for each pixel, and the tone curve control step processing, Causing the computer to execute processing including a saturation control step of controlling the depth value controlled by the tone curve based on the saturation.

  In one aspect of the present invention, for each pixel of the input image, a high frequency component is extracted as focus information from the luminance signal of the peripheral pixels, and the integrated focus information is obtained by integrating the focus information of the peripheral pixels for each pixel. For each of the pixels, the dark portion focus information of the input image is generated by replacing the focus information of the surrounding pixels with a value obtained based on focus information larger than its own focus information, and the pixels Illumination components are extracted by integrating each luminance signal with the luminance signal of surrounding pixels, and a product-sum operation using a coefficient using the illumination components of the integrated focus information and the dark portion focus information is performed for each pixel. The depth value is calculated, the depth value for each pixel is normalized, the normalized depth value is controlled by a tone curve, the saturation is detected for each pixel, and is controlled by the tone curve Serial depth value is controlled by the saturation.

  The image processing apparatus of the present invention may be an independent apparatus or a block that performs image processing.

  According to one aspect of the present invention, the depth value of a subject in an image can be accurately obtained.

[Configuration example of image processing apparatus]
FIG. 1 is an image processing apparatus showing a configuration example of an embodiment to which the present invention is applied.

  The image processing apparatus 1 in FIG. 1 includes a YUV separation unit 11, an integration unit 12, a sharpness control unit 13, a contrast control unit 14, a saturation control unit 15, and a depth detection unit 16. The image processing apparatus 1 obtains a spread in the image from the input image, that is, a depth value in units of pixels, and performs a process corresponding to the obtained depth value on the input image to improve the image quality.

  The YUV separation unit 11 converts, for example, an input image composed of RGB signals into YUV (brightness and color difference signals) in units of pixels, and separates the luminance signal Y into an integration unit 12, a sharpness control unit 13, and a contrast. It supplies to the control part 14 and the depth detection part 16. The YUV separation unit 11 supplies the color difference signals U and V separated from the input image to the saturation control unit 15 and the depth detection unit 16. When the input image is composed of YUV signals, the YUV separation unit 11 merely separates the luminance signal Y and the color difference signals U and V.

  The sharpness control unit 13 generates a luminance signal Ys in which sharpness is controlled by controlling the high frequency component of the input luminance signal Y to be emphasized in units of pixels, and supplies the luminance signal Ys to the integration unit 12.

The contrast control unit 14 performs control so as to enhance the contrast of the input luminance signal Y in units of pixels, thereby generating the luminance signal Yc with enhanced contrast and supplying the luminance signal Yc to the integration unit 12.

  The saturation control unit 15 generates the color difference signals Uc and Vc in which the saturation is emphasized by multiplying the color difference signals U and V by a predetermined coefficient for each pixel, and supplies them to the integration unit 12.

  The depth detection unit 16 obtains a depth value d for each pixel from the luminance signal Y and the color difference signals U and V, and supplies the depth value d to the integration unit 12. The depth detection unit 16 will be described in detail later with reference to FIG.

  Based on the depth value d, the integration unit 12 integrates the luminance signals Y, Yc, Ys and the color difference signals Uc, Vc so as to emphasize perspective, and outputs the result as an output image P ′.

[Configuration example of depth detector]
Next, a detailed configuration example of the depth detection unit 16 will be described with reference to FIG.

  The depth detection unit 16 includes a focus information extraction unit 31, a region integration unit 32, a dark processing unit 33, a depth generation unit 34, an illumination component extraction unit 35, and a saturation detection unit 36, and includes a luminance signal Y and a color difference. The depth value d is obtained in units of pixels from the signals U and V and supplied to the integration unit 12.

  The focus information extraction unit 31 extracts a high frequency component by multiplying the luminance signal Y by a horizontal HPF (High Pass Filter) and a vertical HPF, and supplies it as focus information F to the region integration unit 32 and the dark part processing unit 33. The configuration example of the focus information extraction unit 31 will be described in detail later with reference to FIG.

  The region integration unit 32 is configured by an LPF (Low Pass Filter), and smoothes the focus information F so as to bring the focus information F closer to the value of the focus information F of the surrounding pixels for each pixel. FL is generated and supplied to the depth generation unit 34. The region integration unit 32 only needs to be able to perform integration processing so that the focus information F is brought close to the value of the peripheral focus information F. Therefore, in addition to the LPF, the region integration unit 32 can perform FIR (Finite Impulse Response Filter), IIR ( Infinite Impulse Response Filter), ε filter, or median filter may be used.

  The dark part processing unit 33 obtains an average value larger than the focus information F of the target pixel by comparing with the focus information F of the peripheral pixel for each pixel. Then, the dark part processing unit 33 generates focus information FD by processing the dark part to be bright by replacing the focus information F of the target pixel with the obtained average value, and supplies the focus information FD to the depth generation unit 34.

  The illumination component extraction unit 35 is configured by an LPF. The illumination component extraction unit 35 extracts the illumination component LF by smoothing the luminance signal Y so as to approach the luminance signal Y of the surrounding pixels, and the depth generation unit 34. To supply.

  The saturation detection unit 36 detects the saturation S from the color difference signals U and V, and supplies the detected saturation S to the depth generation unit 34.

  The depth generation unit 34 generates and outputs a depth value d based on the focus information FL, FD, the illumination component LF, and the saturation S. The depth generator 34 will be described in detail later with reference to FIG.

[Configuration example of focus information extraction unit]
Next, a detailed configuration example of the focus information extraction unit 31 will be described with reference to FIG.

  The focus information extraction unit 31 includes a horizontal direction HPF 51, an absolute value processing unit 52, a maximum value extraction unit 53, a vertical direction HPF 54, and an absolute value processing unit 55. Output as information F.

  The horizontal HPF 51 is, for example, an HPF that extracts a high-frequency component in the horizontal direction as shown in the left part of FIG. 4, and extracts the high-frequency component YH in the horizontal direction of the input luminance signal Y to perform absolute value processing. To the unit 52.

  The absolute value processing unit 52 obtains the absolute value of the high frequency component YH extracted as the high frequency component in the horizontal direction and supplies it to the maximum value extracting unit 53.

  The vertical HPF 54 is an HPF that extracts a high-frequency component in the vertical direction as shown in the right part of FIG. 4, for example, and extracts the high-frequency component YV in the vertical direction of the input luminance signal Y to perform absolute value processing. To the unit 52.

  The absolute value processing unit 55 obtains the absolute value of the high frequency component YV extracted as the high frequency component in the vertical direction and supplies it to the maximum value extracting unit 53.

  The maximum value extraction unit 53 extracts the maximum values of the high frequency components YH and YV and outputs them as focus information F.

[Configuration example of depth generator]
Next, a detailed configuration example of the depth generation unit 34 will be described with reference to FIG.

  The depth generation unit 34 includes a light / dark integration unit 71, a normalization unit 72, a tone curve control unit 73, and a saturation control unit 74, and is based on the focus information FL, FD, the illumination component LF, and the saturation S. The depth value d is generated and output.

  The light / dark integration unit 71 combines the luminance signal YL from the region integration unit 32 and the luminance signal YD from the dark processing unit 33 at a ratio based on the illumination component LF supplied from the illumination component extraction unit 35. Thus, the bright part and the dark part of the luminance signal are integrated, and the synthesized signal g is supplied to the normalizing unit 72.

  The normalization unit 72 normalizes the synthesized signal g supplied from the light / dark integration unit 71 and supplies the normalized signal to the tone curve control unit 73 as a normalized synthesized signal g ′.

  The tone curve control unit 73 generates a depth value dg by controlling the synthesized signal g ′ according to a preset tone curve, and supplies the depth value dg to the saturation control unit 74.

  The saturation control unit 74 multiplies the depth value dg supplied from the tone curve control unit 73 by the coefficient set based on the saturation S supplied from the saturation detection unit 36 to obtain the depth value d. Generate and supply to the integration unit 12.

[Image processing by image processing device]
Next, image processing by the image processing apparatus 1 in FIG. 1 will be described with reference to the flowchart in FIG.

  In step S11, the YUV separation unit 11 determines whether a new image has been supplied, and repeats the same processing until a new image is supplied. In step S1, for example, when a new image is input, the process proceeds to step S12.

In step S12, the YUV separation unit 11 converts and separates the YUV signal into pixel units, and supplies the luminance signal Y to the integration unit 12, the sharpness control unit 13, the contrast control unit 14, and the depth detection unit 16. The color difference signals U and V are supplied to the saturation control unit 15 and the depth detection unit 16.

  In step S <b> 13, the sharpness control unit 13 controls the luminance signal Y of the input image so as to enhance the sharpness by pixel enhancement in units of pixels, and supplies the luminance signal Ys to the integration unit 12.

  In step S14, the contrast control unit 14 controls the luminance signal Y of the input image so as to increase the contrast in units of pixels by the arithmetic processing represented by the following formula (1), and generates and integrates the luminance signal Yc. To the unit 12.

Yc = (Y−Ymin) / (Ymax−Ymin)
... (1)

  In equation (1), Y is the luminance signal of each pixel of the input image, Ymin is the minimum value of the luminance signal Y in the input image, and Ymax is the maximum value of the luminance signal Y in the input image. , Yc are brightness signals subjected to contrast control.

  That is, by the process of step S14, the contrast control unit 14 converts the luminance signal Y into a ratio of the difference between the luminance signal Y and the minimum value Ymin to the difference between the maximum value Ymax and the minimum value Ymin. To emphasize.

  In step S15, the saturation control unit 15 performs control so as to enhance the saturation by multiplying each of the supplied color difference signals U and V by a predetermined coefficient κ, and the color difference signal Uc (= κU). ), Vc (= κV) is supplied to the integration unit. The predetermined coefficient κ can be set arbitrarily.

  In step S <b> 16, the depth detection unit 16 performs a depth detection process based on the luminance signal Y and the color difference signals U and V, obtains the depth value d in units of pixels, and supplies it to the integration unit 12.

[Depth detection processing]
Here, the depth detection processing will be described with reference to the flowchart of FIG.

  In step S <b> 31, the saturation detection unit 36 calculates the saturation S from the supplied color difference signals U and V by calculating the following equation (2), and supplies it to the depth generation unit 34.

S = √ (U ² + V ² )
... (2)

  Here, U and V are color difference signals. That is, the saturation S is obtained as the square root of the sum of squares of the color difference signals U and V.

  In step S <b> 32, the illumination component extraction unit 35 performs processing so as to approach the luminance signal Y of the surrounding pixels by smoothing with the luminance signal Y around each pixel, and extracts the illumination component LF to extract the depth generation unit. 34. That is, since a high frequency component is small in a bright region even in a focused state, a value close to the luminance signal Y of surrounding pixels is extracted as the illumination component LF.

  In step S <b> 33, the focus information extraction unit 31 performs focus information extraction processing for each pixel, obtains focus information F from the luminance signal Y, and supplies the focus information F to the region integration unit 32 and the dark part processing unit 33.

[Focus information extraction processing]
Here, the focus information extraction processing will be described with reference to the flowchart of FIG.

  In step S51, the horizontal HPF 51 subjects the input luminance signal Y to horizontal filter processing as shown in the left part of FIG. 4, for example, and extracts the high frequency component YH to the absolute value processing unit 52. Supply.

  In step S52, the absolute value processing unit 52 calculates the absolute value of the high frequency component YH extracted as the high frequency component in the horizontal direction and supplies the absolute value to the maximum value extracting unit 53.

In step S53, the vertical HPF 54 subjects the input luminance signal Y to vertical filtering as shown in the right part of FIG. 4, for example, and extracts the high frequency component YV to the absolute value processing unit 55 . Supply.

  In step S54, the absolute value processing unit 55 obtains the absolute value of the high frequency component YV extracted as the high frequency component in the vertical direction and supplies the absolute value to the maximum value extracting unit 53.

  In step S <b> 55, the maximum value extraction unit 53 extracts the maximum value of the high frequency components YH and YV, that is, any larger value, and outputs the maximum value to the region integration unit 32 and the dark part processing unit 33.

  Through the above processing, the larger value of the high-frequency components in the horizontal direction or the vertical direction of the luminance signal Y is output as the focus information F for each pixel of the input image P.

  Now, the description returns to the flowchart of FIG.

  In step S <b> 34, the region integration unit 32 smoothes the focus information F, generates the focus information FL close to the value of the focus information F of the surrounding pixels, and supplies the focus information FL to the depth generation unit 34. That is, by this process, the region integration unit 32 smoothes and processes the focus information F, assuming that the entire image is a high-frequency component image, generates the focus information FL, and supplies the focus information FL to the depth generation unit 34.

  In step S <b> 35, the dark part processing unit 33 obtains an average value of the focus information F of the peripheral pixels larger than the focus information F of the pixel to be processed by comparing with the focus information F of the pixel periphery for each pixel. That is, for example, as shown in FIG. 9, the dark part processing unit 33 performs focus information in a one-dimensional range W such as a vertical direction or a horizontal direction centering on a pixel to be processed indicated by a black circle in the drawing. Among F, an average value FA (white circle in FIG. 9) having a value larger than the focus information F of the target pixel indicated by a bold line is obtained. Then, the dark part processing unit 33 generates the focus information FD by replacing the focus information F of the pixel to be processed with the obtained average value FA. As a result of this processing, the focus information FD is generated by replacing the focus information F of the surrounding pixels with the average value FA of the focus information F of the surrounding pixels that is brighter than itself. That is, for example, pixels that are in a dark range in the image are processed brightly. For the processing of the dark part processing unit 33, an average value in a predetermined range near the pixel to be processed may be used. For example, from the focus information F in a two-dimensional range centered on the pixel to be processed. Alternatively, the focus information F may be replaced with an average value of larger values.

  In step S <b> 36, the depth generation unit 34 executes depth generation processing, generates a depth value d based on the focus information FL, FD, the illumination component LF, and the saturation S, and outputs the depth value d to the integration unit 12.

[Depth generation processing]
Here, the depth generation processing will be described with reference to the flowchart of FIG.

  In step S71, the light / dark integration unit 71 calculates the following equation (3) using the focus information FL and the focus information FD, and combines them at a ratio based on the illumination component LF to generate a combined signal g. Then, the synthesized signal g is supplied to the normalizing unit 72.

g = A * FD + (1-A) * FL
... (3)

  Here, g represents a combined signal, FD represents a dark portion luminance signal, YL represents a bright portion luminance signal, and A is a coefficient determined based on the illumination component LF as shown in FIG. It is. That is, the coefficient A is a value from 0 to 1.0, and takes 1.0 when the illumination component LF is close to 0, and is 1.0 until the illumination component LF reaches a predetermined value. When it becomes larger, the illumination component LF becomes linearly smaller as it becomes larger, and when it exceeds a predetermined value, it becomes zero.

  Therefore, the bright / dark integration unit 71 increases the ratio of the bright portion luminance signal YL to the bright portion luminance signal YL and the dark portion luminance signal YD with respect to an image having a large illumination component LF and overall bright. Synthesize. Conversely, the light / dark integration unit 71 increases the ratio of the dark portion luminance signal YD to the dark portion luminance signal YD and the dark portion luminance signal YD with respect to a dark image as a whole. Synthesize.

  As a result, the focus information F is adjusted so as to follow the luminance signal YL having a high frequency component when the input image P is bright as a whole, and conversely, when the input image P is dark as a whole, the focus information F follows the luminance signal YD subjected to the dark portion processing. Adjusted.

  In step S72, the normalization unit 72 normalizes the synthesized signal g by, for example, performing an operation as shown in the following expression (4), and the normalized synthesized signal g ′ is a tone curve control unit. 73.

g ′ = (g−gmin) / (gmax−gmin)
... (4)

  Here, g ′ is a normalized synthesized signal, g is a synthesized signal before being normalized, gmax is a maximum value among synthesized signals g of each pixel in the input image, and gmin is in the input image. The minimum value of the combined signal g of each pixel is represented.

  In step S <b> 73, the tone curve control unit 73 generates a depth value dg by controlling the composite signal g ′ according to, for example, a tone curve as illustrated in FIG. 12 and supplies the depth value dg to the saturation control unit 74. That is, as shown in FIG. 12, in the preset tone curve, when the composite signal g ′ is near 0 or a value near 1, the depth value dg gradually increases and the composite signal g ′ is 0. In the vicinity of .4, the depth value dg increases sharply. For this reason, when the synthesized signal g ′ is large, the depth value dg is set small, and when the synthesized signal g ′ is small, the depth value dg is set large. Further, the depth value dg becomes a value close to either 0 or 1.0 depending on the magnitude of the composite signal g ′, and the depth value is located at the front or the depth value. Whether or not there is is controlled so that it can be divided relatively clearly.

  In step S74, the saturation control unit 74 uses the coefficient B set by the saturation S shown in FIG. 13 supplied from the saturation detection unit 36 to the depth value dg supplied from the tone curve control unit 73. Is used to generate the depth value d controlled by the saturation S. More specifically, the saturation control unit 74 calculates the depth value d by executing the calculation represented by the following expression (5), and supplies the calculated depth value d to the integration unit 12.

d = B × dg
... (5)

  Here, d is a depth value controlled by the saturation S, B is a coefficient set by the saturation S as shown in FIG. 13, and dg is a depth value before being controlled by the saturation S, respectively. Represents.

  As shown in FIG. 13, the coefficient B takes a value in a range from a preset minimum value Bmin to a maximum value Bmax of the coefficient B, and is between the minimum value Smin and the maximum value Smax of the saturation S in the input image. It is a value that is linearly transformed by. That is, the depth value d is controlled such that the greater the saturation S, the greater the depth value dg before the control is, and conversely, the smaller the saturation S, the depth value dg before the control. Is controlled to be a small value. For this reason, the depth value d increases as the saturation S increases, and the depth can be set to a value that can be clearly identified. The depth value d decreases as the saturation S decreases, and the depth is further identified. It is possible to set a difficult value.

  With the above processing, the depth value d is set based on the synthesis coefficient g, the illumination component LF, and the saturation S that are set according to the brightness of the input image in units of pixels. It becomes possible to calculate.

  Now, the description returns to the flowchart of FIG.

  That is, when the depth generation process of step S36 in the flowchart of FIG. 7 ends, the depth detection process of step S16 in the flowchart of FIG. 6 ends, and the process proceeds to step S17.

  In step S17, the integration unit 12 determines the luminance signal Y of the input image, the luminance signal Ys with enhanced sharpness, the luminance signal Yc with enhanced contrast, and the color difference signal with enhanced saturation based on the depth value d. Uc and Vc are integrated and output. More specifically, the integration unit 12 performs calculations as shown in the following formulas (6) to (8), a new luminance signal Y ′ corresponding to the depth value d, and color difference signals U ′, V. An output image P consisting of 'is generated and output.

Y ′ = (1−α) × Y + α × (β × Ys + γ × Yc)
... (6)
U ′ = (1−α) × U + α × Uc
... (7)
V ′ = (1−α) × V + α × Vc
... (8)

  Here, α is a coefficient set corresponding to the depth value d as shown in the upper part of FIG. 14, and β is a coefficient set corresponding to the depth value d as shown in the middle part of FIG. Γ is a coefficient set corresponding to the depth value d as shown in the lower part of FIG.

  That is, the coefficient α takes a value of 0 to 1.0 and linearly changes in the range from the minimum value dmin to the maximum value dmax in the input image of the depth value d. The coefficient β takes a value from the minimum value βmin to the maximum value βmax of the coefficient β, and linearly changes in the range from the minimum value dmin to the maximum value dmax in the input image of the depth value d. The coefficient γ takes a value from a minimum value γmin to a maximum value γmax of the coefficient γ, and linearly changes in a range from the minimum value dmin to the maximum value dmax in the input image of the depth value d.

  Since the coefficients α, β, and γ all increase as the depth value d increases in pixel units, the luminance signal Y ′ is based on the luminance signal Ys with enhanced sharpness and the luminance signal Yc with enhanced contrast. The impact will increase. Further, the color difference signals U 'and V' are both greatly influenced by the color difference signals Uc and Vc whose saturation is controlled. Therefore, by processing the input image P, the image processing apparatus 1 outputs the output image P ′ having a clear and deep feeling with enhanced contrast and sharpness.

  On the other hand, as the depth value d decreases in units of pixels, the coefficients α, β, and γ all have smaller values. Therefore, the luminance signal Y ′ is more influenced by the luminance signal Y in the input image P. Further, the color difference signals U ′ and V ′ are both greatly affected by the color difference signals U and V in the input image P. For this reason, the image processing apparatus 1 outputs the output image P ′ as an almost intact image with a small process applied to the input image P.

  As a result, according to the depth value d, the input image can be processed while being controlled to an appropriate depth feeling.

  Further, as shown in FIG. 15, the coefficients β and γ are set to different values as the minimum values βmin and γmin and the maximum values βmax and γmax, respectively, so that the luminance signal Ys and the contrast with enhanced sharpness are obtained. The degree of influence of the emphasized luminance signal Yc on the luminance signal Y ′ may be adjusted. FIG. 15 shows an example where the minimum value βmin = 0.2, γmin = 0.1, and the maximum value βmax = 0.6, γmax = 0.4.

  Furthermore, the coefficients β and γ may be defined as γ = 1−β, for example, so that the relationship with the coefficients β and γ may be linked. By setting in this way, sharpness is emphasized. It becomes possible to regulate to some extent the degree of influence of the luminance signal Ys and the luminance signal Yc with enhanced contrast on the luminance signal Y ′.

  According to the present invention, it is possible to appropriately set the depth value in units of pixels in an image based on the luminance signal of the bright part and the dark part and the saturation by the color difference signal. For this reason, for example, it is possible to appropriately set the depth value for a dark portion or a flat portion in the image, and it is possible to perform processing in units of pixels corresponding to the depth value.

  By the way, the series of information processing described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 16 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via the bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  An input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of a processing result to a display device, a program and various data A storage unit 1008 including a hard disk drive for storing data, a LAN (Local Area Network) adapter, and the like, and a communication unit 1009 for performing communication processing via a network represented by the Internet are connected. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a block diagram which shows the structural example of the depth detection part of FIG. It is a block diagram which shows the structural example of the focus information extraction part of FIG. It is a block diagram which shows the structural example of horizontal direction HPF and vertical direction HPF. It is a block diagram which shows the structural example of the depth production | generation part of FIG. 3 is a flowchart illustrating image processing by the image processing apparatus in FIG. 1. It is a flowchart explaining a depth detection process. It is a flowchart explaining a focus information extraction process. It is a figure explaining operation | movement of a dark part process part. It is a flowchart explaining a depth production | generation process. It is a figure explaining depth generation processing. It is a figure explaining a tone curve. It is a figure explaining the relationship between the saturation S and the coefficient B. It is a figure explaining the relationship between depth value d and coefficients (alpha), (beta), and (gamma). It is a figure explaining the other relationship between depth value d and coefficients (beta) and (gamma). And FIG. 11 is a diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 YUV separation part, 12 Integration part, 13 Sharpness control part, 14 Contrast control part, 15 Saturation control part, 16 Depth detection part, 31 Focus information extraction part, 32 Area | region integration part, 33 Dark part processing part , 34 Depth generation unit, 35 Illumination component extraction unit, 36 Saturation detection unit, 51 Horizontal HPF, 52 Absolute value processing unit, 52, 53 Maximum value extraction unit, 54 Vertical HPF, 55 Absolute value processing unit, 71 Brightness Integration unit, 72 normalization unit, 73 tone curve control unit, 74 saturation control unit

Claims (5)

Focus information extracting means for extracting high frequency components as focus information from luminance signals of surrounding pixels for each pixel of the input image;
For each pixel, a region integration unit that generates the integrated focus information by integrating the focus information of surrounding pixels;
For each pixel, a dark part processing unit that generates dark part focus information of the input image by replacing the focus information of peripheral pixels with a value obtained based on focus information larger than its own focus information;
Illumination component extraction means for extracting an illumination component by integrating the luminance signal for each pixel with the luminance signal of surrounding pixels;
Brightness and darkness integration means for calculating a depth value for each pixel by a product-sum operation using a coefficient using the illumination component of the integrated focus information and the dark portion focus information;
Normalizing means for normalizing the depth value for each pixel;
A tone curve control means for controlling the depth value normalized by the normalization means by a tone curve;
Saturation detecting means for detecting saturation for each pixel;
An image processing apparatus comprising: saturation control means for controlling the depth value controlled by the tone curve by the tone curve control means based on the saturation. The perspective control means for controlling luminance and color difference signals for each pixel of the input image by a coefficient based on the depth value controlled by the tone curve by the saturation control means. Image processing device. Sharpness control means for controlling the sharpness of the brightness;
Contrast control means for controlling the brightness contrast;
Color difference control means for controlling a color difference signal for each pixel,
The perspective control means uses the brightness controlled by the sharpness control means and the brightness controlled by the contrast control means in addition to the brightness for each pixel to control the brightness by the coefficient, and the color difference The image processing apparatus according to claim 2, wherein the color difference signal is controlled by a coefficient based on the depth value, using the signal and the color difference signal controlled by the color difference control unit. For each pixel of the input image, a focus information extraction step for extracting a high frequency component as focus information from the luminance signal of the surrounding pixels;
For each pixel, a region integration step for generating the integrated focus information by integrating the focus information of surrounding pixels;
For each pixel, a dark part processing step for generating dark part focus information of the input image by replacing the focus information of peripheral pixels with a value obtained based on focus information larger than its own focus information;
Illumination component extraction step of extracting the illumination component by integrating the luminance signal for each pixel with the luminance signal of surrounding pixels;
A light and dark integration step of calculating a depth value for each pixel by a product-sum operation using a coefficient using the illumination component of the integrated focus information and the dark portion focus information;
A normalization step of normalizing the depth value for each pixel;
A tone curve control step for controlling the depth value normalized by the processing of the normalization step by a tone curve;
A saturation detection step for detecting saturation for each pixel;
An image processing method comprising: a saturation control step of controlling the depth value controlled by the tone curve by the saturation by the processing of the tone curve control step. For each pixel of the input image, a focus information extraction step for extracting a high frequency component as focus information from the luminance signal of the surrounding pixels;
For each pixel, a region integration step for generating the integrated focus information by integrating the focus information of surrounding pixels;
For each pixel, a dark part processing step for generating dark part focus information of the input image by replacing the focus information of peripheral pixels with a value obtained based on focus information larger than its own focus information;
Illumination component extraction step of extracting the illumination component by integrating the luminance signal for each pixel with the luminance signal of surrounding pixels;
A light and dark integration step of calculating a depth value for each pixel by a product-sum operation using a coefficient using the illumination component of the integrated focus information and the dark portion focus information;
A normalization step of normalizing the depth value for each pixel;
A tone curve control step for controlling the depth value normalized by the processing of the normalization step by a tone curve;
A saturation detection step for detecting saturation for each pixel;
A program that causes a computer to execute processing including: a saturation control step of controlling the depth value controlled by the tone curve by the saturation by the processing of the tone curve control step.

Images