JP5121294B2 - Image processing method, image processing apparatus, program, recording medium, and integrated circuit - Google Patents

Description

  The present invention relates to an image processing method, an image processing apparatus, a program, a recording medium, and an integrated circuit that enhance the depth and stereoscopic effect of an image according to a near view area and a distant view area in a two-dimensional image.

In order to display the video displayed on the screen of a large screen FPD (flat panel display) device as a more natural video, a technology for improving the “depth feeling” and the “stereoscopic effect” in the display video has been received from the user. There is a strong demand. On the other hand, 3D televisions using human binocular parallax have been proposed, but there are many cases where special dedicated glasses are required, there is a problem that image dependency is large, or special The problem that it is expensive because it requires a device has been pointed out. Currently, as a selling point for large-screen display devices, a technology that realizes the three-dimensional appearance of the displayed image (video) by smoothing the gradation and color characteristics in the display image of the large-screen display device is appealing. Often done.
Humans use not only binocular parallax but also monocular information such as color information, saturation, brightness, contrast (of color information and brightness information), shadows, texture gradients, relative sizes, etc. It is clear that the image perceives a sense of depth and stereoscopic effect.

As the prior art 1 using such monocular information, the difference between an object and a surrounding background is visually increased by clarifying the difference between a portion having a strong contour component and another portion, and the depth of the image is increased. There is a technique for giving a feeling (see, for example, Patent Document 1).
FIG. 71 is a block diagram showing a configuration of a three-dimensional expression circuit 9000 according to the prior art. As shown in FIG. 71, three-dimensional representation circuit 900 0, the input data consisting of the luminance signal of the input digital video signal, a contour extraction unit 5002 for extracting a contour component from a target pixel and its neighboring pixels, the contour extraction An output data B (corresponding to the signal (data) at point B in FIG. 71 ) is calculated by adding the output of the unit 5002 and the input data (data input at the terminal 5000) that is synchronized with the output. A contour emphasis processing unit 5011 including an adding circuit 5004. Also, three-dimensional representation circuit 900 0, the averaging circuit 5005 for calculating the averaged data A from the target pixel of the input data and its neighboring pixel (corresponding to the signal at the point A in FIG. 71 (data).), Contour extraction A threshold value processing unit 5009 that compares the contour component that is the output of the unit 5002 with an arbitrarily set threshold value, and the averaged data A and the contour emphasizing process of the averaging circuit 5005 according to the magnitude of the output of the threshold value processing unit 5009 A selector that selectively outputs the output data B of the unit 5011.

In this apparatus (three-dimensional expression circuit 900 0 ), in a portion where the contour component is originally strong (the edge amount is large), data in which the contour is emphasized is output to the input data, and the contour component is originally weak (the edge amount is small and flat). Since the data obtained by averaging the input data is output in the portion), the difference between the portion having a strong contour component and the other portion becomes clear, and the image formed by the output data processed by the three-dimensional expression circuit 900 Becomes a three-dimensional image.
Further, as another conventional technique 2, in an image formed by an image signal, a region having a high signal level of the primary differential signal or the secondary differential signal of the image signal is detected as a foreground region of the image, and the signal level is low. There is a method for detecting a region as a distant view region of an image, and adding a sense of perspective by changing the strength of contour enhancement according to the attribute of the detected region (attribute of whether it is a distant view region or a foreground region) (for example, see Patent Document 2) ).

FIG. 72 is a block diagram showing a configuration of a perspective emphasis circuit 9100 according to the prior art.
The perspective emphasis circuit 9100 is an image signal (image signal input to the perspective emphasis circuit 9100) is a luminance signal, and performs edge emphasis by edge-added edge emphasis on an image formed by the image signal. A differentiation circuit 6001 that performs first and second differentiations on an image signal (input image signal), and detection of a perspective area of the image from the first and second differentiation values calculated by the differentiation circuit 6001 A perspective detection unit 6002, a coefficient weighting unit 6003 that multiplies a secondary differential signal by a coefficient value according to the detection result of the perspective detection unit 6002, and an input image signal and a coefficient weighting delayed by a delay unit 6004 to adjust timing An adding unit 6005 for adding the output signal from the unit 6003.
The perspective detection unit 6002 uses a target pixel (a pixel corresponding to the input image signal) based on the signal level of the signal S3 obtained by quantizing the primary differential signal DY1 and the signal level of the signal S5 obtained by quantizing the secondary differential signal DY2. Is a pixel belonging to the foreground area or a pixel belonging to the distant area. The perspective detection unit 6002 compares the signal level of the S3 signal with a set value for determining whether or not the pixel belongs to the contour region, and if the signal level of the S3 signal is equal to or higher than the set value, the S4 signal is 1 ", otherwise, the S4 signal is set to" 0 ". In the region where the S4 signal is “1”, the perspective detection unit 6002 has a signal level of the S5 signal (a signal obtained by quantizing the absolute value of the second-order differential signal DY2) for the pixels in the contour portion. It is determined whether the foreground or the background is based on whether or not the threshold value TH is exceeded. The perspective detection unit 6002 determines that the target pixel belongs to the foreground area when the signal level of the S5 signal is higher than the TH value. Then, the perspective detection unit 6002 designates the coefficient KM to be multiplied by the secondary differential signal DY2 as K1, which is larger than the default value KS, and outputs the designated value to the coefficient weighting unit 6003. On the other hand, the perspective detection unit 6002 determines that the target pixel belongs to the distant view area when TH is smaller than S5. Then, the perspective detection unit 6002 designates the coefficient KM to be multiplied by the secondary differential signal DY2 as a value K2 that is smaller than the default KS, and outputs the designated value to the coefficient weighting unit 6003.

As described above, the perspective emphasis circuit 9100 determines whether the pixel is included in the foreground area or the far-field area based on the threshold value determination of the secondary differential signal DY2 of the pixel that is supposed to exist in the contour portion. Judgment. In the perspective emphasis circuit 9100, the coefficient weighting coefficient for the secondary differential signal DY2 is increased in the case of a foreground and reduced in the case of a distant scene in accordance with the determination result, and the contour is emphasized. , Improve the perspective near the contour.
Japanese Patent Laid-Open No. 10-126708 JP 2004-159148 A

In the conventional image processing method of the first example, the preset threshold signal Y is compared with the contour component value X obtained by contour extraction, and the contour-enhanced data is output or the averaged data is output. To decide. Therefore, it is easily affected by the threshold determination accuracy. Further, in the conventional example 1, since the averaged data is output in the portion where the contour component is weak, blurring (blurring phenomenon) occurs in that portion. Originally, when the image data has a high resolution, the reduction in resolution due to the blur greatly affects the image quality. In particular, in the case of large-screen TVs that are becoming HD, it is preferred to enhance the contour enhancement so as not to emphasize noise because of the demands of users and broadcasts. In such a situation, the contour is weak. Even in a flat part, it is difficult to accept blurring processing such as averaging.
Further, in the above conventional image processing method 2, since it is determined whether the background is a foreground or a background only for an outline having a somewhat large differential value, a weak outline such as a texture pattern, ambient light, etc. For a contour portion that could not be properly extracted due to shooting conditions, a foreground / distant view is not determined. That is, there is a high possibility that the contour extraction by the primary differential signal is affected by the threshold determination accuracy. Furthermore, since the threshold processing is further performed on the secondary differential signal of the pixel that seems to be the contour portion, and it is determined whether the foreground or the foreground is in the background, it is easily affected by the threshold determination accuracy when determining the depth information. Therefore, for example, even if the object is a contour part of the same object and is at the same distance, there is a risk that a part that is strongly emphasized and a part that is weakly applied, or only strong edge enhancement and weak edge enhancement are performed in the contour part. There is a risk that a discontinuous portion of luminance may occur due to not being performed.

In addition, the second derivative signal of luminance is small because of blur due to image shooting conditions (defocus or motion) or interpolation when a low resolution image is simply converted to a high resolution. It is not possible to make a determination as to whether or not Therefore, for example, when an image encoded by an irreversible encoding method is decoded and distortion in the decoded image is removed by a low-pass filter or the like, the entire surface of the image may be changed depending on the setting of the threshold determination value of the secondary differential signal. There is a risk that the image is determined to be a distant view and the original edge enhancement processing is not appropriately performed.
The present invention solves the problems of the conventional example 1 and improves the depth and stereoscopic effect of the image. Thus, the edge enhancement data and the averaged data as in the conventional example 1 are not used separately. Accordingly, the depth correction is realized by performing color correction accordingly. In particular, by correcting a color contrast effect, which is one of human visual characteristics, in conjunction with an effect based on depth information, an image processing apparatus and an image processing that can further enhance the depth of color of interest to humans method, program, and to realize a recording medium and an integrated circuit.

In addition, by using color contrast characteristics and brightness contrast characteristics, which are one of human visual characteristics, even if there is a blurred part on the image, the target part (on the image to be processed) It is possible to determine whether the color and brightness of (part) are large. Furthermore, it has been pointed out that such parts (parts where the color and brightness of the target part are larger than the surroundings) are more likely to be noticed by humans. The depth information in the image can be easily estimated.
The present invention also solves the problem related to depth information estimation in Conventional Example 2, and is estimated based on color contrast characteristics and brightness contrast characteristics without using contour extraction by differential signals as in Conventional Example 2. An image processing apparatus, an image processing method, and an image processing apparatus that acquire an image with an enhanced sense of depth by using a depth information to strengthen a color that is highly interested in humans and weaken a color that is less interesting to humans An object is to realize a program, a recording medium, and an integrated circuit.

1st invention corrects the color information of an image signal based on the color information calculation part which calculates the color information of an image signal from the image signal which can form the image which consists of pixels, and the input depth information A correction amount control unit for obtaining a correction gain for the image, a color contrast amount based on color information of a pixel of interest that is a pixel to be processed in the image signal, color information of peripheral pixels of the pixel of interest, and a correction gain The image processing apparatus includes: a color information correction unit that corrects color information; and an output unit that outputs the color information corrected by the color information correction unit in a predetermined format.
In this image processing apparatus, the color information calculation unit calculates the color information of the image signal from the image signal, and the correction amount control unit corrects the color information of the image signal based on the input depth information. The gain is obtained, and the color information is corrected by the color information correction unit based on the color contrast amount and the correction gain. Then, the output unit outputs the color information corrected by the color information correction unit in a predetermined format.

Accordingly, by performing color correction according to the depth information without using the contour emphasis data or the averaged data thereof, it is possible to improve the sense of depth in the image processed by the image processing apparatus.
Here, the “depth information” is information indicating a degree indicating that a pixel of interest (or a block of interest consisting of a plurality of pixels) that is a pixel to be processed in the image signal is a foreground (or a distant view). For example, from a shooting point (which may be a virtual shooting point, and not necessarily an actual shooting point) from which a two-dimensional image to which the target pixel belongs is acquired to a point in the three-dimensional space corresponding to the target pixel. This corresponds to the information correlated with the three-dimensional distance.
The “predetermined format” refers to a format related to an image to be displayed on the display device. For example, an image (video) format format (for example, JPEG image format, BMP image format, MPEG format, NTSC format). This corresponds to the video format).

The second invention is the first invention, wherein the color information correction unit increases the color information of the pixel of interest and the surrounding pixels of the pixel of interest as the degree of the depth information about the pixel of interest increases. The near-field feeling is enhanced by the color contrast effect with the color information, and correction is performed so that the far-field feeling is enhanced by the color contrast effect as the degree of the depth information regarding the pixel of interest indicating that the distance is large.
Accordingly, by performing color correction according to the depth information without using the contour emphasis data or the averaged data thereof, it is possible to improve the sense of depth in the image processed by the image processing apparatus. Furthermore, since the color contrast effect, which is one of the visual characteristics of human beings, can be color-corrected (color information correction) in conjunction with the effect of depth information, in the image processed by this image processing apparatus, Can enhance the depth of color of interest.

The third invention is the second invention, the color information correction unit, a color characteristic calculating unit for calculating a color contrast amount, based on the correction gain and the color contrast amount, compared for correcting the color information A correction amount calculation unit that calculates a correction coefficient; and a correction unit that corrects color information based on the contrast correction coefficient.
In this image processing apparatus, the color information is corrected by changing the contrast correction coefficient. Therefore, the desired color information correction can be easily realized simply by changing the contrast correction coefficient according to the characteristics to be realized. Can be made.
4th invention is 3rd invention, Comprising: A color characteristic calculation part uses ratio RCi (= Ci / ACi) with the color information Ci of the pixel of interest, and the representative value ACi of the color information of a surrounding pixel as a color contrast amount Calculate as The correction amount calculation unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain . When the color contrast amount RCi is “1”, the contrast correction coefficient KLi is set to “0” and the contrast correction is performed. The contrast correction coefficient KLi is set so that the coefficient KLi is monotonous with respect to the color contrast amount RCi. The correction unit obtains the color information correction amount dCi by controlling the contrast correction coefficient KLi, the color information Ci, and the constant α1, and calculates the corrected color information Ci_new after correcting the target pixel as Ci_new = (
By setting the color information Ci) + (color information correction amount dCi), the color information of the target pixel is corrected.

In this image processing apparatus, since the upper limit value and the lower limit value of the contrast correction coefficient KLi are determined by the depth information by the correction amount control unit, the contrast effect suitable for the pixels in the foreground area is compared with the pixels belonging to the foreground area. The color information correction that realizes the contrast effect suitable for the pixels in the distant view area can be performed on the pixels belonging to the distant view area.
“Monotone” means monotonous increase or monotonic decrease. “The contrast correction coefficient KLi is monotonous with respect to the color contrast amount RCi” indicates that the contrast correction coefficient KLi is monotonously increased or monotonically decreased with respect to the color contrast amount RCi. However, as long as the color contrast amount RCi has a monotonic characteristic as a whole, a portion that does not become monotonous locally may be included.
Note that the color contrast amount may be other than the ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and the representative value ACi of the color information of the peripheral pixels. For example, RCi (= Ci-ACi) You may make it use the value by.

Further, the color information correction amount dCi may be obtained, for example, as dCi = (contrast correction coefficient KLi) × (color information Ci) × α1 (α1: constant), or (Equation 5) and (Equation 6). Alternatively, it may be obtained by (Equation 8) and (Equation 9).
5th invention is 4th invention, Comprising: Color characteristic calculation part WHEREIN: Ratio RSi (= Si / ASi) with saturation Si of a pixel of interest and the representative value ASi of the saturation of a surrounding pixel is a color contrast amount Calculate as
The correction amount calculation unit is a positive value smaller than the first foreground threshold (> 0) when the degree of correction gain for the target pixel is the maximum and the color contrast amount is greater than “1”. The contrast correction coefficient Ki is set to the value, and when the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second foreground threshold (<0), and the correction gain for the target pixel is set. When the degree indicating that the image is a distant view is the maximum, when the color contrast amount is greater than “1”, the positive contrast value is smaller than the first distant threshold value (> 0), which is a value smaller than the first distant view threshold value. When the contrast correction coefficient Ki is set and the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second far-field threshold (<0), which is larger than the second near-field threshold. To do.

The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the constant α2, and the corrected color information Si_new after correction of the target pixel is expressed as Si_new = (saturation Si) + ( By setting the saturation correction amount dSi), the color information of the target pixel is corrected.
In this image processing apparatus, when the degree of the depth information regarding the target pixel is the maximum, that is, when it is determined that the target pixel belongs to the farthest background, for example, the contrast correction coefficient Ki is − The characteristic curve of the contrast correction coefficient Ki with respect to the color contrast amount RSi is set such that (RSi, Ki) = (1, 0) while setting to take a value in the range of Th1 ≦ Ki ≦ Th1 (Th1> 0). As described above, the contrast correction coefficient Ki can be set to increase monotonously with respect to the color contrast amount RSi. Further, in this image processing apparatus, when the degree of the depth information regarding the target pixel is the maximum, that is, when it is determined that the target pixel belongs to the closest background, for example, the contrast correction coefficient Ki is , −Th2 ≦ Ki ≦ Th2 (Th2> 0, Th2> Th1), and the characteristic curve of the contrast correction coefficient Ki with respect to the color contrast amount RSi is (RSi, Ki) = (1 , 0), the contrast correction coefficient Ki can be set to monotonically increase with respect to the color contrast amount RSi.

In this way, since the range that the contrast correction coefficient takes when the target pixel is included in the foreground area can be set larger than when the target pixel is included in the foreground area, the saturation contrast effect of the pixels belonging to the foreground area is strengthened. By reducing the saturation contrast effect of the pixels belonging to the distant view area (decreasing the intensity of the saturation contrast effect compared to the near view), the perspective and depth of the image processed by this image processing device can be adjusted appropriately. Can be improved.
The color contrast amount may be other than the ratio RSi (= Si / ASi) between the color information (saturation) Si of the pixel of interest and the representative value ASi of the color information (saturation) of the peripheral pixels. , RSi (= Si-ASi) may be used.
Further, the saturation correction amount dSi may be obtained as dSi = (contrast correction coefficient Ki) × (saturation Si) × α2 (α2: constant), or obtained by (Equation 5 ) and (Equation 6 ). Alternatively, it may be obtained by (Equation 8 ) and (Equation 9 ). In addition, when using (Formula 5 ), (Formula 6 ), (Formula 8 ), and (Formula 9 ) here, it goes without saying that Ci becomes Si.

A sixth invention is compared to a second aspect, the color information correcting section for determining a color characteristic calculating unit for calculating a color contrast amount, the contrast correction coefficient control amount α4 based on the color contrast amount and the color information A correction amount control unit; a correction amount calculation unit that calculates a contrast correction coefficient for correcting color information based on the correction gain and the color contrast amount; and color information based on the contrast correction coefficient and the contrast correction coefficient control amount. A correction unit that corrects.
In this image processing apparatus, since the contrast correction coefficient control amount α4 is determined by the contrast correction amount control unit based on the color contrast amount and the color information, excessive color correction (color saturation, etc.) and a large decrease in color information are prevented. Can be suppressed. In particular, in this image processing apparatus, by changing the contrast correction coefficient control amount α4, color correction for preventing impression deterioration due to excessive correction in the skin color region, and impression improvement effect in the sky blue region and the green region are exhibited. In addition, strong color correction can be easily performed.

7th invention is 6th invention, Comprising: The color information correction | amendment part further has the brightness characteristic calculation part which calculates a brightness contrast amount based on the brightness information of an attention pixel, and the brightness information of a surrounding pixel. Have. The contrast correction amount control unit obtains a contrast correction coefficient control amount α4 based on the color contrast amount and the brightness contrast amount.
In this image processing apparatus, since the contrast correction coefficient control amount α4 is obtained by the contrast correction amount control unit based on the color contrast amount and the brightness contrast amount, pixels with high human interest can be corrected with higher accuracy. .
The eighth invention is the sixth invention, wherein the color characteristic calculation unit calculates a ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and the representative value ACi of the color information of the surrounding pixels as a color contrast amount. Calculate as The correction amount calculation unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain . When the color contrast amount RCi is “1”, the contrast correction coefficient KLi is set to “0” and the contrast correction is performed. The contrast correction coefficient KLi is set so that the coefficient KLi is monotonous with respect to the color contrast amount RCi. The correction unit obtains the color information correction amount dCi by controlling the contrast correction coefficient KLi, the color information Ci, and the contrast correction coefficient control amount α4, and calculates the corrected color information Ci_new after correcting the target pixel as C
By setting i_new = (color information Ci) + (color information correction amount dCi), the color information of the target pixel is corrected.

Note that the color contrast amount may be other than the ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and the representative value ACi of the color information of the peripheral pixels. For example, RCi (= Ci-ACi) You may make it use the value by.
Further, the color information correction amount dCi may be obtained as dCi = (contrast correction coefficient KLi) × (color information Ci) × (correction coefficient control amount α4), or by (Equation 5) and (Equation 6). You may obtain | require by (Formula 8) and (Formula 9).
The ninth invention is the seventh invention, wherein the color characteristic calculation unit calculates a ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and the representative value ACi of the color information of the surrounding pixels as a color contrast amount. Calculate as The correction amount calculation control unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain , and sets the contrast correction coefficient KLi to “0” when the color contrast amount RCi is “1”. The contrast correction coefficient KLi is set so that the correction coefficient KLi is monotonous with respect to the color contrast amount RCi. The correction unit obtains the color information correction amount dCi by controlling with the contrast correction coefficient KLi, the color information Ci, and the contrast correction coefficient control amount α4. By setting Ci) + (color information correction amount dCi), the color information of the target pixel is corrected.

Note that the color contrast amount may be other than the ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and the representative value ACi of the color information of the peripheral pixels. For example, RCi (= Ci-ACi) You may make it use the value by.
The color information correction amount dCi may be calculated as dCi = (contrast correction coefficient KLi) × (color information Ci) × (correction coefficient control amount α4), or calculated using (Formula 5) and (Formula 6). Alternatively, it may be obtained by (Equation 8) and (Equation 9).
A tenth aspect of the invention is the eighth aspect of the invention, wherein the color characteristic calculation unit calculates a ratio RSi (= Si / ASi) between the saturation Si of the target pixel and the representative value ASi of the saturation of the surrounding pixels as a color contrast amount. Calculate as The correction amount calculation unit is a positive value smaller than the first foreground threshold (> 0) when the degree of correction gain for the target pixel is the maximum and the color contrast amount is greater than “1”. The contrast correction coefficient Ki is set to the value, and when the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second foreground threshold (<0), and the correction gain for the target pixel is set. When the degree indicating that the image is a distant view is the maximum, when the color contrast amount is greater than “1”, the positive contrast value is smaller than the first distant threshold value (> 0), which is a value smaller than the first distant view threshold value. When the contrast correction coefficient Ki is set and the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second far-field threshold (<0), which is larger than the second near-field threshold. To do. The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the contrast correction coefficient control amount α4, and obtains corrected color information Si_new after correction of the target pixel as Si_new = (saturation). By setting (Si) + (saturation correction amount dSi), the color information of the target pixel is corrected.

The color contrast amount may be other than the ratio RSi (= Si / ASi) between the color information (saturation) Si of the pixel of interest and the representative value ASi of the color information (saturation) of the peripheral pixels. , RSi (= Si-ASi) may be used.
Further, the saturation correction amount dSi may be obtained as dSi = (contrast correction coefficient Ki) × (saturation Si) × (correction coefficient control amount α4), or by (Equation 5) and (Equation 6). You may obtain | require by (Formula 8) and (Formula 9). In addition, when using (Formula 5), (Formula 6), (Formula 8), and (Formula 9) here, it goes without saying that Ci becomes Si.
The eleventh invention is the ninth invention, wherein the color characteristic calculation unit calculates a ratio RSi (= Si / ASi) between the saturation Si of the target pixel and the representative value ASi of the saturation of the surrounding pixels as a color contrast amount. Calculate as The correction amount calculation control unit is a positive value smaller than the first near-field threshold (> 0) when the degree of correction gain for the pixel of interest is the maximum and the color contrast amount is greater than “1”. set the contrast correction coefficient Ki to the value, when the color contrast amount is equal to or less than "1", the second foreground threshold sets the contrast correction coefficient Ki to a larger negative value than (<0), the correction for the pixel of interest When the degree of gain indicating that the subject is far is the maximum, and when the color contrast amount is greater than “1”, the positive value is smaller than the first distant threshold (> 0) that is smaller than the first distant threshold. The contrast correction coefficient Ki is set to a negative value greater than the second foreground threshold (<0), which is larger than the second foreground threshold when the color contrast amount is “1” or less. Set. The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the contrast correction coefficient control amount α4, and obtains corrected color information Si_new after correction of the target pixel as Si_new = (saturation). By setting (Si) + (saturation correction amount dSi), the color information of the target pixel is corrected.

Further, the saturation correction amount dSi may be obtained as dSi = (contrast correction coefficient Ki) × (saturation Si) × (correction coefficient control amount α4), or by (Equation 8) and (Equation 9). You may ask for it. In addition, when using (Formula 5), (Formula 6), (Formula 8), and (Formula 9) here, it goes without saying that Ci becomes Si.
In a twelfth aspect , a color information calculating step for calculating color information of an image signal from an image signal capable of forming an image composed of pixels, and correcting the color information of the image signal based on the input depth information. A correction amount control step for obtaining a correction gain for the image, a color contrast amount based on color information of the pixel of interest that is a pixel to be processed in the image signal, color information of peripheral pixels of the pixel of interest, and a correction gain a color information correction step of correcting the color information, an image processing method having a.
As a result, an image processing method having the same effect as that of the first invention can be realized.
In a thirteenth aspect , the computer uses a color information calculation unit that calculates color information of the image signal from an image signal that can form an image composed of pixels, and the color information of the image signal based on the input depth information. Based on a correction amount control unit for obtaining a correction gain for correction , a color contrast amount based on color information of a pixel of interest that is a pixel to be processed in the image signal, and color information of peripheral pixels of the pixel of interest, and a correction gain This is a program for functioning as a color information correction unit that corrects color information.

As a result, it is possible to realize a program that exhibits the same effects as those of the first invention.
In a fourteenth aspect , the computer uses a color information calculation unit that calculates color information of the image signal from an image signal that can form an image composed of pixels, and the color information of the image signal based on the input depth information. Based on a correction amount control unit for obtaining a correction gain for correction , a color contrast amount based on color information of a pixel of interest that is a pixel to be processed in the image signal, and color information of peripheral pixels of the pixel of interest, and a correction gain The computer-readable recording medium stores a program for functioning as a color information correction unit that corrects color information.
Thereby, it is possible to realize a computer-readable recording medium in which a program having the same effect as that of the first invention is recorded.
In a fifteenth aspect , a color information calculation unit that calculates color information of an image signal from an image signal that can form an image composed of pixels, and the color information of the image signal is corrected based on the input depth information. A correction amount control unit for obtaining a correction gain for the image, a color contrast amount based on color information of a pixel of interest that is a pixel to be processed in the image signal, color information of peripheral pixels of the pixel of interest, and a correction gain a color information correction unit for correcting the color information is Bei obtain integrated circuits.

Thus, an integrated circuit that exhibits the same effect as that of the first invention can be realized.
A sixteenth aspect of the invention is the first aspect of the invention, further comprising a user mode selection unit that selects a processing mode according to a user instruction, and a display unit that displays an output from the output unit as an image.
Thereby, in this image processing apparatus, the user can select a processing mode.
In a seventeenth aspect based on the sixteenth aspect , the user mode selection unit selects at least a processing mode including information related to color information correction intensity.
Thereby, in this image processing apparatus, the user can select the intensity of color information correction and change the intensity of color information correction in this image processing apparatus. For example, in this image processing apparatus, “strong mode”, “medium mode”, and “weak mode” are provided as processing modes for setting the intensity of color information correction, and the user selects one of the modes. Thus, the color information correction desired by the user is realized.

According to the present invention, it is possible to improve the sense of depth by performing color correction according to depth information without using different contour emphasis data and averaged data thereof, and in particular, with one of human visual characteristics. An image processing apparatus, an image processing method, an image processing apparatus, a program, and a recording medium that can enhance a sense of depth of a color that is highly interested by humans by correcting a certain color contrast effect in conjunction with the effect of depth information And an integrated circuit can be realized.
In addition, according to the present invention, by using the estimated depth information based on the color contrast characteristic and the brightness contrast characteristic without using the contour extraction based on the differential signal, the color that is highly interested in humans is further strengthened. By weakening colors with low interest level, it is possible to realize an image processing device, an image processing method, an image processing device, a program, a recording medium, and an integrated circuit that acquire an image with improved sense of depth.

Hereinafter, first to seventeenth embodiments as the best mode of the present invention will be described.
In the first embodiment, an image processing apparatus and an image processing method for controlling a color correction effect based on a color contrast effect using a correction control parameter based on depth information will be described.
In the second embodiment, an image processing apparatus and an image processing method for controlling the correction amount due to the color contrast effect in the first embodiment using the color contrast amount and color information will be described.
In the third embodiment, an image processing apparatus and an image processing method for controlling the correction amount by the color contrast effect in the first embodiment by the color contrast amount and the hue of the target pixel will be described.
In the fourth embodiment, an image processing apparatus and an image processing method that also use brightness information when correcting color information according to depth information as compared with the first to third embodiments will be described.
In the fifth embodiment, an image processing apparatus and an image processing method for controlling the correction amount by the color contrast effect in the fourth embodiment by the color contrast amount and the brightness contrast amount will be described.

In the sixth embodiment, an image processing apparatus and an image processing method for controlling the correction amount by the color contrast effect in the fourth embodiment by a plurality of color contrast amounts, brightness, brightness contrast amounts, hues, and color information to be corrected Will be described.
In the seventh embodiment, an image processing apparatus and an image processing method will be described in which a color contrast amount is introduced into a predetermined approximate function to calculate a degree of depth, which is used as a correction gain for color correction for improving the sense of depth.
In the eighth embodiment, an image processing apparatus and an image which select a plurality of pixels having a higher degree of attention than the color contrast amount, estimate a foreground position, and determine a correction gain for color correction for improving the sense of depth around the foreground position. A processing method will be described.
In the ninth embodiment, the foreground determination performed in units of pixels in the eighth embodiment is performed by obtaining a color contrast amount based on the average color information in each divided block, and selecting blocks that satisfy a predetermined condition. An image processing apparatus and an image processing method that are determined to be near-field candidates will be described.

In the tenth embodiment, instead of determining whether or not each block is a foreground candidate in the ninth embodiment, a method of obtaining a correction gain value between the target pixels by the block based on the color contrast amount of each block. An image processing apparatus and an image processing method for introducing and correcting a final correction gain value by synthesizing correction gain values for all blocks will be described.
In the eleventh embodiment, image processing for calculating a degree of depth by introducing two feature amounts of brightness contrast amount and color contrast amount into a predetermined approximate function and determining a correction gain for color correction for improving the sense of depth. An apparatus and an image processing method will be described.
In the twelfth embodiment, a plurality of pixels having a high degree of attention are selected from two feature amounts of brightness contrast amount and color contrast amount, a foreground position is estimated, and color correction for improving a sense of depth is performed with the near view position as a center. An image processing apparatus and an image processing method for determining a correction gain will be described.

In the thirteenth embodiment, the foreground determination performed for each pixel in the twelfth embodiment is performed by calculating the color contrast amount and the brightness contrast amount based on the average color information and the average brightness information in each divided block. An image processing apparatus and an image processing method that are determined and determined that a block that satisfies a predetermined condition is a foreground candidate will be described.
In the fourteenth embodiment, instead of determining whether or not each block is a foreground candidate in the thirteenth embodiment, the correction gain value to the target pixel by that block based on the color contrast amount and the brightness contrast amount of each block An image processing apparatus and an image processing method for obtaining a final correction gain value by combining the correction gain values of all blocks will be described.
In the fifteenth embodiment, a block having a high degree of attention is selected as a foreground candidate block based on three feature amounts of a contrast amount based on average brightness in a block, a contrast amount based on average color information, and a frequency component amount, and a foreground position is estimated. An image processing apparatus and an image processing method for determining a correction gain for color correction for improving the sense of depth centering on the foreground position will be described.

In the sixteenth embodiment, a correction gain value for a target pixel in each block is obtained from three feature amounts of a contrast amount by average brightness in a block, a contrast amount by average color information, and a frequency component amount, and correction by all blocks is performed. An image processing apparatus and an image processing method for obtaining a final correction gain value by combining gain values will be described.
In the seventeenth embodiment, an image processing apparatus and method in which a user selects a processing mode such as a correction level using the image processing apparatuses of the first to sixteenth embodiments as an image processing unit will be described.
[First Embodiment]
From Figure 1 with reference to FIG. 10, a first embodiment of the present invention, an image processing method and image processing apparatus 100 for correcting color information of a pixel will be described in accordance with the depth information.
FIG. 1 shows a configuration of an image processing apparatus 100 according to the first embodiment. FIG. 5 shows the configuration of the color information correction unit 12. FIG. 2 shows a processing flowchart of the image processing method according to the first embodiment. FIG. 6 shows a flowchart of the color information correction unit and a flowchart of color contrast amount data calculation.

The present invention is a device that corrects color information in an image by processing image data. For example, a photographing device such as a digital still camera or a digital video camera, or a digital image acquired by these photographing devices is edited. It is mounted on an image editing apparatus, a mobile phone used in a mobile environment, a car mobile device / PDA, or a large video display device used in various environments.
<1.1: Configuration of Image Processing Device>
The image processing apparatus 100 mainly includes a color information calculation unit 11 that calculates color information vCi from an input image signal vIi, a correction amount control unit 10 that determines a correction gain Gi from depth information fbi, and a correction amount control unit 10. The color information correction unit 12 that corrects the color information vCi output from the color information calculation unit 11 based on the determined correction gain Gi, and the correction color information vCi_new corrected by the color information correction unit 12 is converted into various image formats. And an output unit 13 for outputting.

The color information calculation unit 11 receives the input image signal vIi, calculates color information vCi from the input image signal vIi, and outputs the calculated color information vCi to the color information correction unit 12.
The correction amount control unit 10 receives the depth information fbi and determines the correction gain Gi from the depth information fbi. Then, the correction amount control unit 10 outputs the determined correction gain Gi to the color information correction unit 12.
The color information correction unit 12 acquires correction color information vCi_new obtained by correcting the color information vCi output from the color information calculation unit 11 based on the correction gain Gi determined by the correction amount control unit 10, and the acquired correction color information vCi_new is output to the output unit 13.
As illustrated in FIG. 5, the color information correction unit 12 includes a color characteristic calculation unit 20, a correction amount calculation unit 21, and a correction unit 22.

The color characteristic calculation unit 20 includes a peripheral color information calculation unit 23 and a color contrast amount calculation unit 24.
The color characteristic calculation unit 20 receives the color information vCi, obtains the color contrast amount vRCi from the color information vCi, and outputs the obtained color contrast amount vRCi to the correction amount calculation unit 21.
The peripheral color information calculation unit 23 receives the color information vCi, calculates representative color information ACi around the target pixel Pi from the color information vCi for each target pixel Pi, and is ACi vector data (ACi set data). The vACi is output to the color contrast amount calculation unit 24. Here, the vector data in which the initial letter of the variable such as the color information vCi is “v” indicates that the target pixel Pi has a plurality of pieces of information (data). For example, the color information vCi may be information having two pieces of information (data) of the color difference data Cb and Cr for the target pixel Pi, and the saturation information (data) and the lightness of the target pixel Pi. Information having three types of information (data) including information (data) and hue information (data) may be used.

The color contrast amount calculation unit 24 receives the color information vCi and the representative color information vACi, calculates a color contrast amount vRCi for the color information vCi, and outputs the calculated color contrast amount vRCi to the correction amount calculation unit 21.
The correction amount calculation unit 21 receives the color contrast amount vRCi and the correction gain Gi, calculates a correction amount for correcting the color information vCi based on the color contrast amount vRCi and the correction gain Gi, and calculates the calculated correction amount. Output to the correction unit 22.
The correction unit 22 receives the color information vCi and the correction amount calculated by the correction amount calculation unit 21 as input, corrects the color information vCi based on the correction amount calculated by the correction amount calculation unit 21, and displays the corrected color information. The corrected color information vCi_new is output to the output unit 13. Specifically, for example, when the correction unit 22 corrects the saturation Si as the color information,
dSi = Ki (RSi, GSi) × Si × α
Si_new = Si + dSi
To obtain the correction color information Si_new,
When correcting hue Hi as color information,
dHi = Li (RHi, GHi) × Hi × α
Hi_new = Hi + dHi
As a result, the correction color information Hi_new is acquired. Here, Ki (RSi, GSi) and Li (RHi, GHi) are contrast correction coefficients, and α is a predetermined constant.

The output unit 13 converts the corrected color information vCi_new corrected by the color information correction unit 12 into various image formats and outputs the image format.
<1.2: Operation of Image Processing Device>
An overview of the processing of the image processing apparatus 100 will be described with reference to the processing flowcharts of FIGS. 2 and 6.
Image data having a pixel value Ii (vector data of the pixel value Ii is vIi) in the pixel i (referred to as a pixel to be processed, hereinafter also referred to as “target pixel”) is input to the image processing apparatus 100. When input, the color information calculation unit 11 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Here, the input image data is HSV space data composed of hue H, saturation S, and lightness V that can easily handle color information (or YCbCr space data composed of luminance Y, color difference Cb, Cr, lightness L , La * b * space data composed of colors a * and b * ). However, it is also possible to handle input image data as it is. That is, the input image data may be converted into color information vCi composed of, for example, three data of hue H, saturation S, and brightness V. Further, the input image data may be converted into, for example, color information Ci including only the hue H (in this case, the color information Ci is single data and not vector data).

On the other hand, the depth information fbi in the pixel i is input to the correction amount control unit 10. There are many formats and expressions as examples of the depth information fbi in the pixel i. For example, the one shown in FIG. 3 can be considered. As shown in FIG. 3A, it is assumed that there are images of two cars with different senses of depth. The depth information fbi in the pixel i takes a value from “0.0” to “1.0”, and the closer to “1.0”, the closer to the near scene.
FIG. 3B shows an example in which the depth information value of each pixel at the pixel position of the image of FIG. 3A is displayed by mapping to the corresponding pixel position.
For example, as shown in FIG. 3B, the depth information fbi is set to “1.0”, assuming that the pixel in the triangular area before the pixel s included in the foreground car is located in the foreground. . Further, the depth information fbi is set to “0.1” on the assumption that the pixels in the triangular area above the pixel t included in the far view car are located in the far view. Further, fbi = 0.1 is set on the assumption that the pixels in the triangular regions on the left and right sides are also located in the distant view. On the other hand, assuming that the depth information changes linearly in the pixel in the center rectangle, for example, the distance LEN of the line segment connecting the pixel s and the pixel t and the distance between the target pixel u and the pixel s are l. In this case, the depth information fbi of the pixel u is set as shown in (Formula 1). In (Expression 1), fbs indicates depth information at the pixel s, fbt indicates depth information at the pixel t, and fbu indicates depth information at the pixel u.

The depth information fbi prepared according to the input image data vIi defined in this way is input to the correction amount control unit 10 separately from the input image. The depth information can be defined in various ways. In addition, there are many possible ways of inputting. The correction amount control unit 10, by the depth information fbi, determines a correction gain GCi for corrected Ci in color information vCi pixel i (the correction gain simply also denoted child as "Gi".) . Then, the correction amount calculator 21 determines a correction amount for the color information vCi based on the correction gain GCi and the color contrast amount vRCi.
For example, when the input image data vIi is converted into HSV system data (HSV space data) by the color information calculation unit 11, the color information correction unit 12 corrects the saturation S based on the saturation contrast amount. When doing so, the correction amount control unit 10 calculates the contrast correction gain GSi related to the saturation S. When the color information correction unit 12 corrects the hue H based on the color relative ratio amount, the correction amount control unit 10 calculates a contrast correction gain GHi for the hue H.

FIG. 4 is a schematic diagram for explaining an example of a correction amount determination method based on the contrast correction gain value.
FIG. 4B shows the depth information fbi shown in FIG. FIG. 4C shows a contrast correction gain for controlling the correction amount of the saturation Si and the hue Hi corrected by the color contrast effect based on the depth information fbi obtained by the correction amount control unit 10. It is a figure for demonstrating the calculation method of value GSi and GHi. Note that the pixel position in FIG. 4C corresponds to the pixel position in FIG.
FIG. 4C shows that, in the case of saturation correction, the correction effect due to saturation contrast is strengthened as the foreground becomes closer. In addition, the farther the background, the weaker the correction effect due to the saturation contrast. The same applies to the case where the hue is corrected based on the effect of the color relative ratio. By doing so, it is possible to control the color contrast effect based on the depth information and correct the saturation, hue, and the like, compared to the case where the saturation and hue are corrected only with depth information. Thereby, in the image (video) processed by the image processing apparatus 100, it is possible to improve the sense of depth and perspective that is more sharp and visually natural.

The color characteristic calculation unit 20 calculates a color contrast amount RCi for the color information Ci to be corrected among the color information Ci constituting the color information vCi obtained by the color information calculation unit 11. For example, in the case of color information vCi obtained by converting the input image vIi into the HSV system, when the color information Ci to be corrected is saturation S, characteristic information regarding the saturation S is calculated by the color characteristic calculation unit 20. The Here, the color information to be corrected among the color information vCi in the pixel Pi is Ci.
The color characteristic calculation unit 20 performs the processing shown in the flowchart in the right diagram of FIG. That is, the peripheral color information calculation unit 23 and the color contrast amount calculation unit 24 calculate the color contrast amount RCi for each pixel. There are many possible color contrast amounts, but it is preferable to use information corresponding to human visual characteristics in order to realize correction that is closer to the human appearance. Here, there are many human visual characteristics, but the color contrast characteristics related to the color characteristics are used, but in addition to this, brightness contrast and size (depending on a plurality of pixels having similar characteristics) The size of a region to be formed), pixel characteristics, and the like can also be used.

FIG. 7 schematically shows the concept of color contrast characteristics.
In FIG. 7, two circles of red and blue with a central portion opened are drawn in a light yellow background. In addition, the center area | region of two circles shall be the same light yellow as a background. When a person observes these circles by paying attention to the position of the line of sight shown in the star in FIG. 7, the person tends to feel the center of the red circle a little blue, and the center of the blue circle a little red There is a tendency to feel. This phenomenon is apparent from the visual psychology and is a phenomenon caused by the color contrast characteristic. The color contrast characteristic refers to an influence on the saturation and hue of the target object when different saturations and hues surround the target object. And the characteristic shown to the following (1) and (2) is clarified.
(1) When a hue different from the hue of the target object surrounds the target object, it is felt that a complementary color of the surrounding hue is added to the target object.
(2) When the saturation surrounding the target object is higher than the saturation of the target object, the saturation of the target object is felt low. Conversely, when the saturation surrounding the target object is lower than the saturation of the target object, the saturation of the target object is felt high.

  In the present invention, by correcting the color information of the pixels using this color contrast characteristic, an impression closer to human vision is given (when a human views an image (video), the human visual characteristics naturally (Feel) Perform image correction. For example, when the saturation Si is set as the correction target Ci from the color information vCi, it is considered that an impression close to human vision can be promoted by performing the saturation correction as schematically shown in FIG. That is, as shown in the left circle of FIG. 8, the target pixel Pi is in the center of the circle, and the saturation Si of Pi is the representative saturation ASi (for example, the surrounding pixels) of the pixels (a plurality of pixels) around the target pixel Pi. Is lower than the (weighted) average value), the saturation Si of the target pixel Pi is reduced to achieve good image correction (color information correction). Conversely, when the saturation Si of the target pixel Pi is higher than the representative saturation ASi of the pixels (a plurality of pixels) around the target pixel Pi as in the right circle, the saturation Si of the target pixel Pi is increased. By doing so, good image correction (color information correction) is realized. By performing the processing as described above, preferable image (video) correction (color information correction) is realized in terms of human visual characteristics. As the representative saturation SiR of the pixels around the target pixel Pi, for example, the weighted average saturation in the pixel region Ωi having a predetermined area corresponding to the human visual field region is used as the pixel surrounding the target pixel Pi. It is preferable to set the representative saturation ASi, but besides this, for a pixel in the field of view, a histogram (saturation histogram) regarding the saturation of the pixel in the region is obtained, and in the saturation histogram, It is also possible to set the representative saturation SiR to the most saturated saturation value, the representative saturation obtained by clustering based on the statistical distribution in the visual field area, the average saturation in the visual field area, etc. It is. As described above, when color information correction by saturation is performed, color information other than saturation at each pixel Pi is stored (not changed), thereby maintaining the balance of the processed (color information corrected) image. This enables natural color information correction in terms of visual characteristics.

  Further, when the hue Hi is selected as the correction target Ci from the color information vCi, it is considered that the impression close to human vision can be promoted by performing the hue correction as schematically shown in FIG. That is, the target pixel Pi is at the center of the circle as in the left circle of FIG. 9, and the representative hue AHi (for example, the surrounding pixels) of the pixel (a plurality of pixels) around the target pixel Pi is higher than the hue Hi of the target pixel Pi. When the (weighted) average value of the hue of red is red (the hue is closer to the red direction), moving the hue Hi of the target pixel Pi in the blue direction realizes good image correction (color information correction). The At this time, color information other than the hue is stored (not changed). In addition, regarding the hue, in order to suppress the adverse effects that occur on the image by abruptly changing the hue, the hue balance (color balance) can be reduced as much as possible by suppressing the movement amount (change amount) of the hue. Can be held. That is, by suppressing the movement amount (change amount) of the hue, it is possible to prevent the image color from becoming unnatural in terms of visual characteristics. Conversely, when the hue Hi of the target pixel Pi is bluer than the representative hue AHi of the pixels around the target pixel Pi (the hue is closer to the blue direction) as in the right circle, the hue Hi of the target pixel Pi is set to red. By moving in the direction, good image correction (color information correction) is realized. As the representative hue AHi of the pixels around the target pixel Pi, the weighted average hue in the pixel area Ωi having a predetermined area corresponding to the human visual field area is set as the representative hue HSi of the pixels around the target pixel Pi. However, in addition to this, for pixels in the visual field region, a histogram (hue histogram) for the hue of the pixels in the region is obtained, and in the hue histogram, the most frequently used hue value, A representative hue obtained by clustering processing based on a statistical distribution in the visual field area, an average hue in the visual field area, or the like may be used as the representative hue AHi. Here, the case where the color information correction is performed by moving the hue Hi in a predetermined direction has been described, but the hue can also be moved (changed) by changing the color difference Cb and Cr in the YCbCr space. It is. In this case, qualitatively, a blue component is generated by increasing the color difference Cb component, and a red component is generated by increasing the color difference Cr component.

In the image processing apparatus 100 of the present invention, the processing shown in the flowchart in the left diagram of FIG. 6 is executed. In the image processing apparatus 100, the color correction amount based on the color contrast is controlled using the contrast correction gains GSi and GHi obtained from the depth information fbi, so that humans can naturally feel the sense of depth and perspective in terms of visual characteristics. It can be effectively added to an image.
The color characteristic calculation unit 20 calculates color contrast amount data RCi for each pixel.
First, the peripheral color information calculation unit 23 calculates representative color information ACi of pixels around the target pixel Pi. In this case, an area of about 1/6 to 1/3 of the image (all image areas) is set as a peripheral area (viewing area), and weighted average color information ACi is obtained. The color contrast amount calculation unit 24 calculates the color contrast amount RCi of the pixel Pi. Here, the ratio of the target color information Ci of the target pixel Pi to the representative color information ACi is defined as a color contrast amount RCi, and the color contrast amount RCi based on this definition is calculated by the color contrast amount calculation unit 24. In addition to this, it is also possible to define the color contrast amount (color contrast information data) RCi by (Ci-ACi), and execute the processing in the color characteristic calculation unit 20 using this color contrast amount RCi. .

The color information correction unit 12 uses the contrast correction gain values GSi and GHi of the pixels based on the depth information fbi to control the correction amount based on the color contrast effect as shown in FIG.
(1.2.1: About saturation correction)
FIG. 10 is an explanatory diagram of saturation correction. The upper left diagram of FIG. 10 shows a blue circle with low saturation around the center circle of light blue with high saturation at the near and far positions. That is, in the upper left diagram of FIG. 10, the region A in the distant view and the region B in the foreground both have a saturation Si of the central circle higher than the peripheral representative saturation ASi.
In this case, as shown in the upper left diagram of FIG. 10, the color contrast effect in the region A tends to be stronger than the color contrast effect in the region B due to human visual characteristics. That is, in the area A in the foreground A, the person in the area A in the foreground A is less than the extent to which the person feels that the saturation of the center circle is higher than the actual saturation of the center circle. The degree to which the saturation of the central circle feels higher than the actual saturation of the central circle (the level of the color contrast effect) is greater.

  The color information correction unit 12 first corrects the color information vCi based on the saturation contrast correction gain value GSi obtained from the depth information fbi by the correction amount control unit 10 to generate corrected color information vCi_new. In the color information correction unit 12, the maximum value Kmax (GSi) of the contrast correction coefficient Ki (RSi, GSi) for controlling the saturation correction amount dSi by the saturation contrast amount RSi based on the saturation contrast correction gain GSi and The minimum value Kmin (GSi) and the differential value KSlop (GSi) of the contrast correction coefficient Ki (RSi, GSi) at RSi = 1 are controlled. FIGS. 10A to 10C are graphs showing the relationship between these control amounts and the saturation contrast correction gain GSi. Specifically, FIG. 10A shows the relationship between the saturation contrast correction gain GSi and the minimum value Kmin (GSi) of the contrast correction coefficient Ki (RSi, GSi), and FIG. 10B shows the saturation contrast. The relationship between the correction gain GSi and the maximum value Kmax (GSi) of the contrast correction coefficient Ki (RSi, GSi) is shown in FIG. 10 (c). The contrast correction coefficient Ki (RSi, RSi, Saturation contrast correction gain GSi and RSi = 1) The relationship between the differential value KSlop (GSi) of GSi) is shown. FIG. 10D shows the relationship between the saturation contrast amount RSi and the contrast correction coefficient Ki (RSi, GSi).

Then, using Kmax (GSi), Kmin (GSi), and KSlop (GSi) obtained based on the saturation contrast correction gain GSi, as shown in FIG. Thus, the contrast correction coefficient Ki (RSi, GSi) is obtained by a monotonically increasing function.
(Equation 2) and (Equation 3) show examples of Kmax (GSi), Kmin (GSi), KSlop (GSi), and Ki (RSi, GSi).
Based on this value, the saturation correction amount dSi is defined by, for example, (Formula 4). Here, α is a preset value and satisfies 0.0 ≦ α ≦ 1.0.
In addition, KminMax, KminMin, KmaxMax, KmaxMin, and KSMMax in FIGS. 10A to 10C are also preset values,
-1.0 <KminMax <0.0,
-1.0 <KminMin <0.0,
KminMax> KminMin,

0.0 <KmaxMax <1.0,
0.0 <KmaxMin <1.0,
KmaxMax> KmaxMin,

0.0 <KSMax <γ
Satisfied. γ, a, and b are predetermined positive constants.

THO and TH1 are
TH0 = Kmin / KSlop + 1,
TH1 = Kmax / KSlop + 1,
TH1>1.0>TH0> 0.0
Satisfied.

(Equation 3) represents the ratio of the saturation correction amount dSi to the saturation Si when the saturation contrast correction gain value is GSi and the saturation contrast amount is RSi, and RSi is “1”. It means that the larger the value is (the higher the saturation of the target pixel is than the surrounding representative saturation), the larger the value of the contrast correction coefficient Ki.
Further, (Equation 2) means that the contrast correction coefficient Ki that determines the saturation correction amount by the comparison in (Equation 3) greatly varies as the saturation contrast correction gain value GSi increases. As a result, the saturation correction amount dSi proportional to the contrast correction coefficient can also be varied greatly. In other words, (Equation 2) indicates that the greater the saturation contrast correction gain value GSi is (the closer the scene is), the more the correction effect by the saturation contrast (the correction effect in the direction of increasing saturation and the correction effect in the direction of decreasing). Means to strengthen).

By causing the image processing apparatus 100 to execute the above-described processing, appropriate color information correction for saturation is realized, and a human being is naturally present in an image (video) processed by the image processing apparatus 100 in terms of visual characteristics. It is possible to effectively add a sense of depth and sense of perspective.
( 1.2.2 : Others)
In the present invention, when the correction amount by color contrast is controlled using the contrast correction gain value GCi as the correction of the color information Ci (color information correction) in the color information correction unit 12, in addition to the one described above ( It is also possible to realize color information correction by the processing expressed by Equations 5 ) and 6 ).
In (Formula 5 ), Ci is correction target color information of the pixel i, ACi is representative color information around the pixel Pi, and α is a positive constant.

In (Expression 6 ), dTH0 and Gmax are predetermined positive constants, and μ is a predetermined positive constant that satisfies 0 <μ <1.

In this (Equation 5 ), Ci / ACi corresponds to the color contrast information data RCi. GF (x) is not limited to (Equation 6 ), but instead of (Equation 6 ), a non-linear function that is smooth and continuous and monotonically increases in accordance with a change in RCi is used instead of (Equation 6 ). May be.
As the correction amount dCi based on the color contrast, the correction amount dCi according to (Equation 7 ) is adopted, and the processing of the present invention is executed with the correction amount dCi according to (Equation 7 ), so that the color information Ci of the correction target is abrupt. It is possible to suppress the saturation phenomenon due to the change.

Furthermore, in the present invention, the color contrast information data RCi is defined by (Ci-ACi), and the correction amount dCi based on the color contrast is converted by a predetermined function shown in (Formula 8 ) and (Formula 9 ). By using the obtained values, it is also possible to execute correction of the color information Ci of the target pixel Pi.
In (Formula 8 ) and (Formula 9 ),
dTH1> 0,
dTH2> 0,
Fmax> 0.0,
Fmin <0.0,
0 <α <1.0
Is a constant satisfying.

In the present invention, the upper and lower limits of the correction amount can be set by executing the processing according to (Equation 8 ) and (Equation 9 ), and the color contrast information RCi can be linearly changed within a predetermined range. it can. (Equation 8 ) and (Equation 9 ) are not limited to these, and the processing of the present invention may be executed using other linear functions or nonlinear functions.

Here, parameters such as positive constants used in (Equation 1) to (Equation 9 ) are held as table data, and by executing control according to the parameters in accordance with the present invention by external input or user instruction, It is also possible to change the degree of color correction in the present invention.
Finally, the output unit 13 may output the correction value of the target color information Ci of each pixel Pi obtained by the color information correction unit 12 and other color information values as output image data as they are. Further, the correction value of the target color information Ci of each pixel Pi obtained by the color information correction unit 12 and other color information values are converted into an image format that can be handled by the device used, and output as output image data. You may make it do.
Further, in the above description, the processing of the present invention when the target color information Ci is one (in the case of saturation alone and the case of hue alone) has been described. For example, in the present invention, two processes of saturation and hue It is also possible to execute color information correction by combining color information correction. In this case, color characteristic information data from each color information is obtained separately, and the color information correction unit 12 sets a correction function suitable for the corresponding color information.

In the above description, a case has been described in which the present invention executes a method of calculating the correction amount GCi based on the depth information fbi and correcting the correction amount based on the calculated comparison correction gain GCi and the color contrast amount. In addition, for example, the present invention may be realized by a method of executing color information correction by combining a color correction amount based on depth information fbi and a color correction amount based on color contrast.
Note that (Equation 3) may use one smoothly non-linear function as in (Equation 10 ). Here, g is a predetermined positive constant.

[Second Embodiment]
An image processing method and an image processing apparatus for correcting color information in a pixel according to depth information will be described as a second embodiment of the present invention with reference to FIGS . 11 to 13 .
<2.1: Configuration of Image Processing Device>
The image processing apparatus according to the second embodiment has the same configuration as that of the image processing apparatus 100 according to the first embodiment, and the color information correction unit 12 of the image processing apparatus 100 includes the image processing apparatus according to the present embodiment. The difference is that only the color information correction unit 2012 is replaced. In the image processing apparatus according to the present embodiment, the same parts as those of the image processing apparatus 100 are denoted by the same reference numerals and description thereof is omitted. The image processing apparatus according to the present embodiment is the same as that shown in FIG. 1 except that the color information correction unit 12 is replaced with the color information correction unit 2012. The illustration about is omitted.

FIG. 11 shows the configuration of the color information correction unit 2012 of the image processing apparatus according to the second embodiment of the present invention. FIG. 12 shows a process flowchart of an image processing method according to the second embodiment (an image processing method executed by the image processing apparatus according to the present embodiment).
The image processing apparatus according to the present embodiment is an apparatus that corrects color information in an image by processing image data. The image processing apparatus according to the present embodiment includes, for example, photographing apparatuses such as a digital still camera and a digital video camera, an image editing apparatus that edits a digital image acquired by these photographing apparatuses, a mobile phone used in a mobile environment, It is installed in car mobile devices, PDAs, etc., or large-sized video display devices used in various environments.
The image processing apparatus and the image processing method according to the second embodiment according to the second embodiment are characterized in that, in the first embodiment, a correction coefficient control amount α that determines a correction amount by color contrast, which is a constant number, In the embodiment, the correction coefficient control amount α is not a constant (constant value), but is controlled by the color contrast amount RCi and the correction target color information Ci.

The color information correction unit 2012 acquires the correction color information vCi_new obtained by correcting the color information vCi output from the color information calculation unit 11 based on the correction gain Gi determined by the correction amount control unit 10, and the acquired correction color information vCi_new is output to the output unit 13.
As illustrated in FIG. 11 , the color information correction unit 2012 includes a color characteristic calculation unit 20, a contrast correction amount control unit 30, a correction amount calculation unit 21, and a correction unit 2022.
The correction amount calculation unit 21 receives the color contrast amount vRCi and the correction gain Gi, calculates a correction amount for correcting the color information vCi based on the color contrast amount vRCi and the correction gain Gi, and calculates the calculated correction amount. The data is output to the correction unit 2022.
The contrast correction amount control unit 30 receives the color information vCi and the color contrast amount vRCi, obtains the contrast correction coefficient control amount αi based on the color information vCi and the color contrast amount vRCi, and uses the calculated contrast correction coefficient control amount αi. The data is output to the correction unit 2022.

The correction unit 2022 receives the color information vCi, the correction amount calculated by the correction amount calculation unit 21, and the contrast correction coefficient control amount αi obtained by the contrast correction amount control unit 30, and is calculated by the correction amount calculation unit 21. The color information vCi is corrected based on the corrected amount and the contrast correction coefficient control amount αi, and the corrected color information is output to the output unit 13 as corrected color information vCi_new.
<2.2: Operation of Image Processing Device>
Processing in the image processing apparatus according to the present embodiment will be described based on the processing flowchart of FIG .
Processing from when image data is input to the image processing apparatus according to the present embodiment until the data of each pixel is converted into predetermined color information vCi is the same as in the first embodiment. Further, the processing up to the calculation of the color contrast amount RCi in the pixel i is the same as in the first embodiment, and is omitted.

In the color characteristic calculation unit, the color information Ci to be corrected is calculated from the color information Ci constituting the color information vCi acquired by the color information calculation unit 11. In the contrast correction amount control unit 30, the contrast correction coefficient control amount αi (vector data) is obtained from the color contrast amount RCi acquired by the color characteristic calculation unit 20 and the color information Ci (vCi in the case of vector data) to be corrected. In the case of (vαi), vαi) is calculated.
In the correction amount calculation unit 21, based on the contrast correction gain value GCi obtained from the depth information and the color contrast amount RCi, a contrast correction coefficient (for example, Ki (RSi, GSi) for saturation, Li (RHi for hue). , GHi).).
The correction unit 2022 corrects the color information Ci to be corrected based on the comparison correction amount calculated by the correction amount calculation unit 21 and the comparison correction coefficient αi calculated by the comparison correction amount control unit 30. That is, the contrast correction amount control unit 30 controls the correction coefficient control amount αi for controlling the correction amount dCi (= (contrast correction coefficient) × (correction target color information Ci) × αi)). FIG. 13 schematically shows this control. Note that FIG. 13 shows the case where the saturation Si is to be corrected, but the present invention is not limited to this. For example, the same applies to the color information Ci that can use the color contrast such as hue correction. The idea can be applied to the present invention.

First, in FIG. 13 , defα is set as a reference value (default value) of the correction coefficient control amount α. Further, the color contrast amount RCi is defined by the ratio of the color information Ci to be corrected (the color information Ci of the target pixel Pi) to the color information ACi representing the periphery (the peripheral pixels of the target pixel Pi). In that case, it is necessary to consider the following points (P1) and (P2).
(P1) When the color contrast amount RCi> 1.0 and the color information Ci to be corrected is high (the value of the color information Ci is large), the color contrast correction amount dCi2 tends to be excessively large. As a result, there is a possibility that a saturation phenomenon may occur in the obtained color correction amount Ci_new. Therefore, in this case, the correction coefficient control amount α (the correction coefficient control amount in the target pixel Pi may be expressed as αi, When the pixel Pi is not conscious, it is necessary to suppress the correction coefficient control amount α). (Expression 11 ) shows an example of an expression for calculating the correction coefficient control amount α for executing the suppression process.

  (P2) When the color contrast amount RCi <1.0 and the color information Ci to be corrected is low (the value of the color information Ci is small), the reduction amount of the correction amount dCi2 due to the color contrast becomes large, resulting in image quality degradation. It can happen. In this case, it is necessary to suppress the correction coefficient control amount α. (Expression 11) shows an example of an expression for calculating the correction coefficient control amount α for executing the suppression process.

Here, dA1 is a positive constant and is a small value satisfying dA1 <defα. THCi is a threshold relating to the color information Ci. In the present invention, the correction coefficient control amount α may be controlled by more finely separating the color information Ci. Also, the contrast amount RCi may be divided by providing a plurality of thresholds other than “1.0”, and α may be controlled in the subdivision area. In FIG. 13 and (Equation 11 ), the correction coefficient control amount α is constant in the two-dimensional region composed of the divided Ci and ACi (in the right diagram of FIG. 13 , all four divided regions are However, the present invention is not limited to this. For example, the correction coefficient control amount α is changed continuously according to two dependent variables Ci and ACi. It may be set. Further, in (Equation 11 ), by connecting the boundary surfaces of the divided regions by changing the correction coefficient control amount α for color correction by color contrast so as to be continuous at the boundaries of the divided regions shown in FIG. It is also conceivable to improve the problem of color correction that may occur near the boundary of the divided areas.

In the correction unit 2022, the correction amount calculation unit 21 uses the contrast correction gain value GCi calculated from the depth information fbi by the correction amount control unit 10 and the color contrast amount vRCi calculated by the color characteristic calculation unit 20. The correction amount dCi is obtained using the contrast correction coefficient and the correction coefficient control amount α calculated by the contrast correction amount control unit 30 as described above, and the obtained correction amount dCi is used as color information Ci before correction. Is added to the color information Ci_new after correction. That is, the correction unit 2022
Ci_new = Ci + dCi
Perform the following process.
The output unit 13 may output the correction value of the target color information Ci of each pixel Pi obtained by the color information correction unit 12 and other color information values as output image data as they are. Further, the correction value Ci of the target color information of each pixel Pi obtained by the color information correction unit 12 and other color information values are converted into an image format that can be handled by the device used and output as output image data. You may make it do.

As described above, in the image processing apparatus according to the present embodiment, the correction amount for correcting only the target color information of each pixel by the color contrast is controlled by the color information Ci and the color contrast amount RCi. The overcorrection that may occur in the image processing apparatus 100 can be improved. Furthermore, in the image processing apparatus according to the present embodiment, color correction (color information correction) that improves the depth perceived naturally by humans in terms of visual characteristics in the image processed by the image processing apparatus according to the present embodiment. Can be realized.
In the above description, the processing of the present invention in the case where the target color information Ci is one has been described. However, for example, color information correction may be executed by combining two color information corrections of saturation and hue. Is possible. In this case, color characteristic information data from each color information is obtained separately, and the color information correction unit 12 sets a correction function suitable for the corresponding color information.

In the above description, the correction gain GCi based on the depth information fbi is calculated, and the method of correcting the correction amount based on the calculated correction gain GCi and the color contrast amount is described in the present invention. For example, the present invention may be realized by a method of executing color information correction by combining the color correction amount based on the depth information fbi and the color correction amount based on the color contrast.
[Third Embodiment]
With reference to FIGS. 14 to 18, as a third embodiment of the present invention, an image processing method and image processing apparatus 300 corrects the color information of a pixel will be described in accordance with the depth information.
<3.1: Configuration of Image Processing Device>
FIG. 14 shows a configuration diagram of the image processing apparatus 300. FIG. 15 is a configuration diagram of the hue-corresponding color information correction unit 42.

The image processing apparatus 300 mainly corrects from the color information calculation unit 11 that calculates the color information vCi from the input image signal vIi, the hue information calculation unit 40 that calculates the hue information Hi from the input image signal vIi, and the depth information fbi. Based on the correction amount control unit 10 for determining the gain Gi, the correction gain Gi determined by the correction amount control unit 10 and the hue information Hi output from the hue information calculation unit 40, the color information calculation unit 11 outputs the gain Gi. It includes a hue-corresponding color information correcting unit 42 that corrects the color information vCi, and an output unit 13 that converts the corrected color information vCi_new corrected by the hue-corresponding color information correcting unit 42 into various image formats and outputs it.
Note that in the image processing apparatus 300 according to the third embodiment, the same portions as those of the above-described embodiment are denoted by the same reference numerals, and description thereof is omitted.
The hue information calculation unit 40 receives the input image signal vIi, calculates the hue information Hi from the input image signal vIi, and outputs the calculated hue information Hi to the hue corresponding color information correction unit 42.

As shown in FIG. 11 , the hue-corresponding color information correction unit 42 includes a color characteristic calculation unit 20, a contrast correction amount control unit 3030, a correction amount calculation unit 21, and a correction unit 3022. Note that the color characteristic calculation unit 20 and the correction amount calculation unit 21 are the same as those in the above-described embodiment, and thus description thereof is omitted.
The contrast correction amount control unit 3030 receives the hue information Hi calculated by the hue information calculation unit 40 and the color contrast amount vRCi calculated by the color characteristic calculation unit 20, and based on the color contrast amount vRCi and the hue information Hi. The contrast correction coefficient control amount αi is obtained, and the obtained contrast correction coefficient control amount αi is output to the correction unit 3022.
The correction unit 3022 receives the color information vCi, the contrast correction coefficient calculated by the correction amount calculation unit 21, and the contrast correction coefficient control amount αi output from the contrast correction amount control unit 3030, and controls the contrast correction coefficient and the contrast correction coefficient control. Based on the amount αi, the color information vCi is corrected, and the corrected color information is output to the output unit 13 as corrected color information vCi_new.

<3.2: Operation of Image Processing Device>
FIG. 16 shows a process flowchart of an image processing method (image processing method in the image processing apparatus 300) according to the third embodiment. The image processing apparatus 300 is an apparatus that corrects color information in an image by processing image data. The image processing apparatus 300 is, for example, a photographing device such as a digital still camera or a digital video camera, an image editing device that edits a digital image acquired by these photographing devices, a mobile phone used in a mobile environment, a car mobile device, It is mounted on a PDA or the like, or a large video display device used in various environments.
The image processing apparatus 300 and the image processing method according to the third embodiment are characterized in that the correction coefficient control amount α for determining the correction amount based on the color contrast, which is a fixed number in the first embodiment, is the color contrast amount RCi and the hue. It is in the point controlled by Hi. An outline of processing of the image processing apparatus 300 will be described based on the processing flowchart of FIG .

  The processing until image data is input to the image processing apparatus 300 and the data of each pixel is converted into predetermined color information vCi is the same as in the above-described embodiment. Further, the process up to the calculation of the color contrast amount RCi in the pixel i is the same as that in the above-described embodiment, and the description thereof is omitted. In the present embodiment, the hue information Hi is obtained by the hue information calculation unit 40 from the color information vCi. The hue information Hi is information indicating the type of color (red, blue, green, etc.), and as the hue information Hi, for example, an H component (hue H component) in HSV space conversion can be used. . The hue information Hi can also be defined by Cb and Cr in the YCbCr space. Here, the case where the hue H in the HSV space conversion is used as the hue information Hi will be described below. In the case of this HSV space, the hue of the pixel i has a value from 0 degrees to 360 degrees. For example, the pure red color is expressed near 113 degrees, the pure green color is expressed near 225 degrees, and the pure blue color is expressed near 353 degrees, but there is no clear boundary between the colors. In the present invention, with respect to the hue Hi, an area where the hue is Hh0 to Hh1 degree is a skin color area, an area where the hue is Hs1 to Hs2 degree is a sky blue area, and an area where the hue is Hg1 to Hg2 degree is green. It shall be defined as an area. The areas of skin color, sky blue, and green cannot be determined unambiguously, but an area that includes the corresponding color or a color close to it is set.

The color characteristic calculation unit 20 calculates color information Ci to be corrected from among the color information Ci constituting the color information vCi acquired by the color information calculation unit 11. The contrast correction amount control unit 3030 calculates a contrast correction coefficient control amount αi (vαi in the case of vector data) from the color contrast amount RCi acquired by the color characteristic calculation unit 20 and the hue information Hi.
In the correction amount calculation unit 21, based on the contrast correction gain value GCi obtained from the depth information and the color contrast amount RCi, a contrast correction coefficient (for example, Ki (RSi, GSi) for saturation, Li (RHi for hue). , GHi).).
The correction unit 3022 corrects the color information Ci to be corrected based on the contrast correction amount calculated by the correction amount calculation unit 21 and the contrast correction coefficient αi calculated by the contrast correction amount control unit 3030. That is, the contrast correction amount control unit 3030 controls the correction coefficient control amount αi for controlling the correction amount dCi (= (contrast correction coefficient) × (correction target color information Ci) × αi)). FIG. 17 schematically shows this control. Note that FIG. 17 shows an example of correcting the saturation Si, but the present invention is not limited to this. For example, the same idea is applied to the color information Ci that can use the color contrast such as hue correction. Can be applied to the present invention.

First, in FIG. 17 , defα is set as a reference value (default value) of the correction coefficient control amount α. Further, the color contrast amount RCi is defined by the ratio of the color information Ci to be corrected (the color information Ci of the target pixel Pi) to the color information ACi representing the periphery (the peripheral pixels of the target pixel Pi). In that case, it is necessary to consider the following points (Q1) to (Q3).
(Q1) When the hue Hi is included in the skin color area, the correction in the skin color area greatly affects the user (in terms of visual characteristics, the color change in the skin color area has high sensitivity, and the skin color area is slightly The user can easily recognize the color change.) Therefore, it is necessary to suppress the correction coefficient control amount α in order to suppress overcorrection in the skin color region.
(Q2) When the hue Hi is included in the green or sky blue region, there is a strong tendency for the user to correct the color in a direction that makes the color clearer or brighter. In the case of the sky blue region or the green region, the color contrast amount RCi is in the vicinity of “1.0”, and thus the correction amount is small in the correction processing expressed by (Formula 5) and the like. Therefore, when the color contrast amount RCi is RCi> 1.0, it is necessary to increase the correction coefficient control amount α.

(Q3) When the hue Hi is included in the green or sky blue region, there is a strong tendency for the user to correct the color in a direction that makes the color clearer or brighter. In the case of the sky blue region or the green region, the color contrast amount RCi is in the vicinity of “1.0”, and thus the correction amount is small in the correction processing expressed by (Formula 5) and the like. Therefore, when the color contrast amount RCi is RCi <1.0, it is preferable to slightly reduce the correction coefficient control amount α.
FIG. 17 shows the control of the contrast correction coefficient control amount α when the hue information Hi is included in the skin color area. In the control of the contrast correction coefficient control amount α, control is performed so as to be defα at the boundary between the skin color and another hue and to decrease α toward the center Hhc of the skin color region. (Expression 12 ) shows an example of an expression for calculating the correction coefficient control amount α for executing the suppression process. Note that Hhc = (Hh1 + Hh2) / 2. Dskin is a positive constant.

FIG. 18 shows the control of the correction coefficient control amount α when the hue Hi is included in the sky blue region. The regions represented by the upper equation and the lower equation of these two regions (Equation 13 ). ) In which the value of the correction coefficient control amount α is changed according to the color contrast amount RCi. (Expression 13 ) is an expression for calculating the correction coefficient control amount α in the case of the green region (Hg1 <Hi <Hg2).

(Expression 14 ) is an expression for calculating the correction coefficient control amount α in the case of the sky blue region (Hs1 <Hi <Hs2).

Here, dA2 and dA3 are positive constants and are values satisfying dA2 <defα and dA3 <defα. RC1 and RC2 are values that determine the boundary between the green region and the intermediate region that connects the region with a low color contrast amount and the region within the sky blue region,
0.0 <RC1 <1.0,
1.0 <RC2 <RCMax
Satisfied. Here, RCMax is the maximum color contrast amount.
In the present invention, the correction coefficient control amount α may be controlled by finely separating the hue Hi in the flesh color area, sky blue area, and green area. In the present invention, it is also possible to control the correction coefficient control amount α by dividing the region in the skin color region according to the color contrast amount RCi. In the present invention, the correction coefficient control amount α may be controlled by further subdividing the color contrast amount RCi in the sky blue region and the green region.

In FIG. 18 , there are discontinuous boundaries (the boundary surface where RCi = RC1 and the boundary surface where RCi = RC2). However, the boundary is smoothly changed according to the hue information Hi and the color contrast amount RCi. It is also possible to control the value of the correction coefficient control amount α so as to be a perfect boundary. Further, in (Equation 12 ) and (Equation 13 ), by using a function that smoothly continues on the boundary surface where RCi = RC1 and the boundary surface where RCi = RC2, in the present invention, in the vicinity of these boundaries It is possible to improve the problems that may occur.
Using such a correction coefficient control amount α, a correction amount dCi based on the color contrast RCi controlled by the contrast correction gain value GCi based on the depth information fbi is obtained, and added to the color information Ci before correction, thereby correcting the color information. Ci_new is obtained. That is, in the correction unit 3022 ,
Ci_new = Ci + dCi
Perform the following process.

Since the processing in the output unit 13 is the same as that in the above-described embodiment, the description thereof is omitted.
As described above, in the image processing apparatus 300, the correction amount for correcting only the target color information of each pixel by the color contrast is controlled by the hue information Hi and the color contrast amount RCi, so that the image processing according to the above-described embodiment is performed. Overcorrection that may occur in the apparatus can be improved. Furthermore, the image processing apparatus 300 can realize color correction (color information correction) that improves the sense of depth that a human feels naturally in terms of visual characteristics in an image processed by the image processing apparatus 300. In addition, the image processing apparatus 300 does not generate a blur region due to a decrease in resolution as in the prior art, and can suppress a large image quality deterioration due to a decrease in resolution.
In the above description, the processing of the present invention in the case where the target color information Ci is one has been described. However, for example, color information correction may be executed by combining two color information corrections of saturation and hue. Is possible. In this case, the color characteristic information data from each color information is obtained separately, and the hue-corresponding color information correction unit 42 sets a correction function suitable for the corresponding color information. Further, in the image processing apparatus 300, when the hue information Hi is included in the target color information Ci, the process of the hue information calculation unit 40 may be performed as it is, but the process of the hue information calculation unit 40 is not performed. The hue information Hi calculated by the color information calculation unit 11 may be used.

In the above description, the correction gain GCi based on the depth information fbi is calculated, and the method of correcting the correction amount based on the calculated correction gain GCi and the color contrast amount is described in the present invention. For example, the present invention may be realized by a method of executing color information correction by combining the color correction amount based on the depth information fbi and the color correction amount based on the color contrast.
[Fourth Embodiment]
With reference to FIGS. 19 to 22, as a fourth embodiment of the present invention, an image processing method and image processing apparatus 400 for correcting color information of a pixel will be described in accordance with the depth information.
<4.1: Configuration of Image Processing Device>
FIG. 19 shows a configuration diagram of an image processing apparatus 400 according to the fourth embodiment of the present invention. FIG. 20 shows a configuration diagram of the brightness-corresponding color information correction unit 52.

The image processing apparatus 400 mainly includes a color information calculation unit 11 that calculates color information vCi from an input image signal vIi, a brightness information calculation unit 50 that calculates brightness information Yi from the input image signal vIi, and depth information fbi. Based on the correction amount control unit 10 for determining the correction gain Gi from the correction amount Gi, the correction gain Gi determined by the correction amount control unit 10 and the brightness information Yi output from the brightness information calculation unit 50. A brightness-corresponding color information correction unit 52 that corrects the color information vCi output from the output unit 13, and an output unit 13 that converts the correction color information vCi_new corrected by the brightness-corresponding color information correction unit 52 into various image formats and outputs them. Consists of
Note that in the image processing apparatus 400 according to the fourth embodiment, portions similar to those of the above-described embodiments are denoted by the same reference numerals, and description thereof is omitted.
The brightness information calculation unit 50 receives the input image signal vIi, calculates brightness information Yi from the input image signal vIi, and outputs the calculated brightness information Yi to the brightness corresponding color information correction unit 52.

As shown in FIG. 20 , the brightness-corresponding color information correction unit 52 includes a color characteristic calculation unit 20, a contrast correction amount control unit 4030, a correction amount calculation unit 21, and a correction unit 4022. Note that the color characteristic calculation unit 20 and the correction amount calculation unit 21 are the same as those in the above-described embodiment, and thus description thereof is omitted.
The contrast correction amount control unit 4030 receives the brightness information Yi calculated by the brightness information calculation unit 50 and the color contrast amount vRCi calculated by the color characteristic calculation unit 20, and inputs the color contrast amount vRCi and the brightness information Yi. Based on this, the contrast correction coefficient control amount αi is obtained, and the obtained contrast correction coefficient control amount αi is output to the correction unit 4022.
The correction unit 4022 receives the color information vCi, the contrast correction coefficient calculated by the correction amount calculation unit 21, and the contrast correction coefficient control amount αi output from the contrast correction amount control unit 4030, and performs contrast correction coefficient and contrast correction coefficient control. Based on the amount αi, the color information vCi is corrected, and the corrected color information is output to the output unit 13 as corrected color information vCi_new.

<4.2: Operation of Image Processing Device>
FIG. 21 shows a process flowchart of an image processing method (image processing method in the image processing apparatus 400) according to the fourth embodiment. The image processing apparatus 400 is an apparatus that corrects color information in an image by processing image data. The image processing apparatus 400 is, for example, a photographing device such as a digital still camera or a digital video camera, an image editing device that edits a digital image acquired by these photographing devices, a mobile phone used in a mobile environment, a car mobile device, It is mounted on a PDA or the like, or a large video display device used in various environments.
The image processing apparatus 400 and the image processing method according to the fourth embodiment are characterized in that the correction coefficient control amount α for determining the correction amount based on the color contrast, which is a fixed number in the first embodiment, is the color contrast amount RCi and the brightness. It is in the point which is controlled by the information Yi. Based on the processing flowchart of FIG. 21, an outline of processing of the image processing apparatus 400 will be described.

The processing from when the image data is input to the image processing apparatus 400 until the data of each pixel is converted into the predetermined color information vCi is the same as in the previous embodiment. Further, the process up to the calculation of the color contrast amount RCi in the pixel i is the same as that in the above-described embodiment, and thus the description thereof is omitted. In the present embodiment, the brightness information calculation unit 50 obtains brightness information Yi from the input image data vIi. There are many examples of brightness information, such as brightness Y of YCbCr space data composed of brightness Y, color difference Cb, Cr, and La * b * space data composed of brightness L, color a *, b *. The lightness L and the like are typical examples. Here, the case where the luminance Y in the YCbCr space data is calculated by the pixel i and used as the brightness information Yi will be described.
The contrast correction amount control unit 4030 calculates the correction coefficient control amount α based on the brightness information Yi and the color contrast amount RCi. The image processing apparatus 400 can generate a more desired image by changing the correction coefficient control amount α.

FIG. 21 shows a processing flow of color information correction processing realized by controlling the correction coefficient control amount α in the image processing apparatus 400. Here, defα is a reference value (default value) of the correction coefficient control amount α. Further, the color contrast amount RCi is defined by the ratio of the color information Ci to be corrected (the color information Ci of the target pixel Pi) to the color information ACi representing the periphery (the peripheral pixels of the target pixel Pi). In that case, it is necessary to consider the following points (R1) and (R2).
(R1) When the value of the brightness information Yi of the target pixel i is high (large), it may be felt that the effect resulting from color correction is excessive color correction in terms of human visual characteristics. In particular, when the color contrast amount RCi is RCi> 1.0, it is preferable to suppress the correction coefficient control amount α.
(R2) When the brightness information Yi value of the target pixel i is low (small), there is a possibility that the effect produced as a result of color correction is not felt so much. In particular, when the color contrast amount RCi <1.0, the color information correction effect by suppressing the color information from the surroundings due to the color contrast effect is low because the brightness information is low (that is, the brightness of the area around the target pixel). It is difficult to feel because it is dark. Therefore, when the value of the brightness information Yi is low (small) and the color contrast amount RCi <1.0, it is preferable to increase the correction coefficient control amount α. On the other hand, when the value of the brightness information Yi is low (small) and the color contrast amount RCi> 1.0, the correction coefficient control amount α is preferably set to a reference value or a slightly larger value. (Equation 15 ) is an example of an equation for calculating the correction coefficient control amount α in consideration of the above concept.

Here, dA3 and dA3s are positive constants, and are small values that satisfy dA3 <defα, dA3s <α, and dA3s <dA3. Yh is a threshold value for determining whether the brightness information Yi is large, and may be divided by a plurality of Yh. Similarly, the color contrast amount RCi may be divided by providing a plurality of thresholds other than “1.0”. In FIG. 22 and (Equation 15 ), the correction coefficient control amount αi is set so that the two-dimensional region composed of the divided Yi and ACi is constant, but is limited to this. For example, the correction coefficient control amount α may be set so as to change continuously according to two dependent variables Yi and ACi. Further, in (Equation 15 ), by changing the correction coefficient control amount α for color correction by color contrast so as to be continuous at the boundaries of the divided areas shown in FIG. It is also conceivable to improve the problem of color correction that may occur near the boundary of the divided areas.

As described above, in the image processing apparatus 400, the correction amount for correcting only the target color information of each pixel by the color contrast is controlled by the brightness information Yi and the color contrast amount RCi, whereby the image according to the above-described embodiment. Overcorrection that may occur in the processing apparatus can be improved. Further, the image processing apparatus 400 can appropriately perform color information correction even in a low-brightness area (dark area) where the effect of color information correction is small in the image processed by the image processing apparatus 400, and the depth. Color correction (color information correction) that improves the feeling can be performed. Further, the image processing apparatus 400 does not generate a blur region due to a reduction in resolution as in the prior art, and can suppress a large deterioration in image quality due to a reduction in resolution.
In the above description, the processing of the present invention in the case where the target color information Ci is one has been described. However, for example, color information correction may be executed by combining two color information corrections of saturation and hue. Is possible. In this case, the color characteristic information data from each color information is obtained separately, and the brightness-corresponding color information correction unit 52 sets a correction function suitable for the corresponding color information.

In the above description, the correction gain GCi based on the depth information fbi is calculated, and the method of correcting the correction amount based on the calculated correction gain GCi and the color contrast amount is described in the present invention. For example, the present invention may be realized by a method of executing color information correction by combining the color correction amount based on the depth information fbi and the color correction amount based on the color contrast.
[Fifth Embodiment]
With reference to FIGS. 23 to 27, a fifth embodiment of the present invention, an image processing method and an image processing apparatus will be described for correcting color information of a pixel in accordance with depth information.
<5.1 Configuration of Image Processing Device>
FIG. 23 shows a configuration of the brightness-corresponding color information correction unit 5052 of the image processing apparatus according to the fifth embodiment of the present invention. Note that the image processing apparatus according to this embodiment according to the fifth embodiment has the same configuration as the image processing apparatus 400 according to the fourth embodiment, and the brightness-corresponding color information correction unit of the image processing apparatus 400. The only difference is that 52 is replaced with a brightness-corresponding color information correction unit 5052.

Further, in the image processing apparatus according to this embodiment according to the fifth embodiment, the same parts as those of the above-described embodiment are denoted by the same reference numerals, and description thereof is omitted.
As shown in FIG. 23 , the brightness-corresponding color information correction unit 5052 includes a color characteristic calculation unit 20, a brightness characteristic calculation unit 60, a contrast correction amount control unit 5030, a correction amount calculation unit 21, and a correction unit 5022. And comprising. Note that the color characteristic calculation unit 20 and the correction amount calculation unit 21 are the same as those in the above-described embodiment, and thus description thereof is omitted.
The brightness characteristic calculation unit 60 receives the brightness information Yi from the brightness information calculation unit 50, and calculates the brightness contrast amount RYi by comparing the brightness information Yi with the representative brightness information AYi that represents the periphery of the brightness information Yi. Then, the calculated brightness contrast amount RYi is output to the contrast correction amount control unit 5030.
The contrast correction amount control unit 5030 receives the brightness contrast amount RYi calculated by the brightness characteristic calculation unit 60 and the color contrast amount vRCi calculated by the color characteristic calculation unit 20, and receives the color contrast amount vRCi and the brightness contrast amount. Based on RYi, a contrast correction coefficient control amount αi is obtained, and the obtained contrast correction coefficient control amount αi is output to the correction unit 5022.

The correction unit 5022 receives the color information vCi, the contrast correction coefficient calculated by the correction amount calculation unit 21, and the contrast correction coefficient control amount αi output from the contrast correction amount control unit 5030, and performs contrast correction coefficient and contrast correction coefficient control. Based on the amount αi, the color information vCi is corrected, and the corrected color information is output to the output unit 13 as corrected color information vCi_new.
<5.2: Operation of Image Processing Device>
FIG. 25 shows a process flowchart of an image processing method according to the fifth embodiment (an image processing method in the image processing apparatus according to the present embodiment). FIG. 26 is a process flowchart illustrating the process of calculating the brightness contrast amount in the process performed by the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment is an apparatus that corrects color information in an image by processing image data. The image processing apparatus according to the present embodiment includes, for example, photographing apparatuses such as a digital still camera and a digital video camera, an image editing apparatus that edits a digital image acquired by these photographing apparatuses, a mobile phone used in a mobile environment, It is installed in car mobile devices, PDAs, etc., or large-sized video display devices used in various environments.

The image processing apparatus and the image processing method according to the fifth embodiment according to the fifth embodiment are characterized in that, in the first embodiment, a correction coefficient control amount α for determining a correction amount based on the color contrast, which is a fixed number, is used as the color contrast. The control is based on the amount RCi and the brightness contrast amount RYi. An outline of processing of the image processing apparatus according to the present embodiment will be described based on the processing flowcharts of FIGS . 25 and 26 .
The processing from when the image data is input to the image processing apparatus according to the present embodiment until the data of each pixel is converted into the predetermined color information vCi is the same as in the previous embodiment. In addition, since the processing for calculating the color contrast amount RCi in the pixel i is the same as that in the above-described embodiment, the description thereof is omitted. In the present embodiment, the brightness information Yi obtained from the color information vCi by the brightness information calculation unit 50 is the same as in the above-described embodiment, and thus the description thereof is omitted.
The brightness characteristic calculation unit 60 calculates the brightness contrast amount RYi by comparing the brightness information Yi with the representative brightness information AYi representing the periphery thereof. As shown in the flowchart of FIG . 26 and the component diagram of FIG . 24 , the brightness contrast amount RYi is generated for each pixel through the ambient brightness information calculation unit 70 and the brightness contrast amount calculation unit 71. Many brightness contrast amounts are conceivable, but in order to realize correction that is closer to the human appearance, information corresponding to human visual characteristics is used. There are many human visual characteristics, and the image processing apparatus according to this embodiment of the present embodiment uses the brightness contrast amount in consideration of the brightness contrast characteristics related to the brightness characteristics. It is known that the color contrast with respect to the color also occurs in the brightness information, and the brightness contrast amount is a numerical value of the degree. For example, consider an example in which there is a small center circle with higher brightness than the surrounding area at the center of a large circle with low brightness. In this case, it is clear from visual psychology that the center of a large circle tends to feel brighter than actual brightness. This phenomenon is a phenomenon caused by brightness contrast characteristics, and occurs when the brightness of an object is affected by the brightness of the surroundings when different brightness surrounds the target object.

That is, when the brightness surrounding the object is higher than the brightness of the object itself, the brightness of the object is felt low. Conversely, when the brightness surrounding the object is lower than the brightness of the object itself, the brightness of the object is felt high.
In the present invention, this brightness contrast characteristic is combined with the color contrast characteristic used in the above-described embodiment, so that the color correction of a portion that is more highly human-interesting than the image processing apparatus according to the above-described embodiment is performed. In addition, the result is combined with color correction based on depth information. Thereby, in the present invention, it is possible to generate an image that improves the sense of depth that humans feel naturally.
In the image processing apparatus according to the present embodiment, about 1/6 to 1/3 of the image corresponding to the human visual field is used as the peripheral region when obtaining the brightness contrast amount RYi, as in the case of the color contrast. Processing is performed using the occupied pixel range (region). At this time, as the representative brightness AYi of the pixels around the target pixel Pi, for example, the weighted average brightness in the pixel area Ωi having a predetermined area corresponding to the human visual field area is set as the surroundings of the target pixel Pi. It is preferable to use the representative brightness AYi of the pixel of the pixel, but in addition to this, for a pixel in the viewing area, a histogram (luminance histogram) is obtained for the brightness (luminance) of the pixel in the area, In the luminance histogram, the representative brightness AYi is represented by the most frequent luminance value, the representative luminance obtained by the clustering process based on the statistical distribution in the visual field area, the average luminance degree in the visual field area, and the like. It is also possible to do.

In addition, as a definition of the brightness contrast amount RYi,
(1) The ratio of the brightness information Yi to be corrected to the brightness information AYi representing the periphery thereof,
(2) A value obtained by subtracting brightness information AYi representing the periphery from brightness information Yi to be corrected,
Etc. can be adopted. In addition, the fact that there are many other definitions is the same as in the case of the color contrast amount RCi.
The contrast correction amount control unit 5030 calculates a correction coefficient control amount α based on the brightness contrast amount RYi and the color contrast amount RCi. Then, by changing the correction coefficient control amount α, it is possible to generate a more desired image in the image processing apparatus according to the present embodiment.
FIG. 27 schematically shows the control of the correction coefficient control amount α. Here, defα is a reference value (default value) of the correction coefficient control amount α. Further, the color contrast amount RCi is defined by the ratio of the color information Ci to be corrected to the color information ACi representing the periphery thereof. Further, the brightness contrast amount RYi is defined as the ratio of the brightness information Yi to be corrected to the brightness information AYi representing the periphery thereof. In this case, it is necessary to consider the following points (S1) to (S3).

(S1) When the brightness contrast amount RYi and the color contrast amount RCi of the target pixel i are both high (large values), the pixel i may be included in a region that is easily noticed. Therefore, in order to obtain more effects, it is better to increase the correction coefficient control amount α.
(S2) When the brightness contrast amount RYi and the color contrast amount RCi of the target pixel i are both low (small values), the pixel i may be included in a region that does not attract much attention. Therefore, in order to suppress the effect, it is better to suppress the correction coefficient control amount α.
(S3) In the region between (S1) and (S2), the value of the correction coefficient control amount α belonging to the middle in both regions is used, or the default value of the correction coefficient control amount α is used.
(Equation 16 ) is an example of an equation for calculating the correction coefficient control amount α based on the above concept. (Expression 16 ) is an expression using the default value defα in the region described in (S3) above.

Here, dA4 is a positive constant and is a small value satisfying dA4 <defα. The color contrast amount RCi and the brightness contrast amount RYi are “1.0”, and the range that can be taken is divided. A plurality of threshold values may be provided and divided. In FIG. 27 and (Equation 16 ), the correction coefficient control amount αi is set so that the two-dimensional region formed by the divided RYi and RCi is constant, but is not limited to this. For example, the correction coefficient control amount α may be set so as to change continuously according to the two dependent variables RYi and RCi. Further, in (Equation 16 ), by changing the correction coefficient control amount α for color correction by color contrast so as to be continuous at the boundaries of the divided areas shown in FIG. It is also conceivable to improve the problem of color correction that may occur near the boundary of the divided areas.

As described above, in the image processing apparatus according to the present embodiment, the correction amount for correcting only the target color information of each pixel by the color contrast is controlled by the brightness contrast amount RYi and the color contrast amount RCi, and the degree of attention is increased. The effect of improving the feeling of depth is further enhanced by enhancing the effect of color correction in a high region and suppressing color correction in a region of low attention. Further, in the image processing apparatus according to the present embodiment, a blur region due to a decrease in resolution as in the conventional technique does not occur, and large image quality deterioration due to a decrease in resolution can be suppressed.
In the above description, the processing of the present invention in the case where the target color information Ci is one has been described. However, for example, color information correction may be executed by combining two color information corrections of saturation and hue. Is possible. In this case, color characteristic information data from each color information is obtained separately, and the brightness-corresponding color information correction unit 5052 sets a correction function suitable for the corresponding color information.

In the above description, the correction gain GCi based on the depth information fbi is calculated, and the method of correcting the correction amount based on the calculated correction gain GCi and the color contrast amount is described in the present invention. For example, the present invention may be realized by a method of executing color information correction by combining the color correction amount based on the depth information fbi and the color correction amount based on the color contrast.
[Sixth Embodiment]
With reference to FIGS. 28 to 30, a sixth embodiment of the present invention, an image processing method and image processing apparatus 600 corrects the color information of a pixel will be described in accordance with the depth information.
<6.1: Configuration of Image Processing Device>
FIG. 28 shows a configuration diagram of an image processing apparatus 600 according to the seventh embodiment of the present invention. FIG. 29 shows a configuration diagram of the extended color information correction unit 80.

The image processing apparatus 600 includes a color information calculation unit 11 that calculates color information vCi from the input image signal vIi, a brightness information calculation unit 50 that calculates brightness information Yi from the input image signal vIi, and a hue from the input image signal vIi. A hue information calculation unit 40 that calculates information Hi and a correction amount control unit 10 that determines the correction gain Gi from the depth information fbi are provided. Further, the image processing apparatus 600 expands the color information vCi output from the color information calculation unit 11 based on the correction gain Gi, the brightness information Yi, and the hue information Hi determined by the correction amount control unit 10. The information correction unit 80 and the output unit 13 that converts the correction color information vCi_new corrected by the extended color information correction unit 80 into various image formats and outputs the image format.
Note that in the image processing device 600 according to the sixth embodiment, the same portions as those in the above-described embodiment are denoted by the same reference numerals, and description thereof is omitted.

As shown in FIG. 29 , the extended color information correction unit 80 includes a color characteristic calculation unit 20, a brightness characteristic calculation unit 60, a contrast correction amount control unit 6030, a correction amount calculation unit 21, a correction unit 6022, Is provided. The color characteristic calculation unit 20, the correction amount calculation unit 21, and the brightness characteristic calculation unit 50 are the same as those in the above-described embodiment, and thus the description thereof is omitted.
The contrast correction amount controller 6030 is calculated by the brightness information Yi calculated by the brightness information calculator 50, the brightness contrast amount RYi calculated by the brightness characteristic calculator 60, and the hue information calculator 40. The hue information Hi and the color information vCi calculated by the color information calculation unit 11 are input, and the contrast correction coefficient control amount αi is obtained based on the brightness information Yi, the brightness contrast amount RYi, and the hue information Hi. The calculated contrast correction coefficient control amount αi is output to the correction unit 6022.

The correction unit 6022 receives the color information vCi, the contrast correction coefficient calculated by the correction amount calculation unit 21, and the contrast correction coefficient control amount αi output from the contrast correction amount control unit 6030, and controls the contrast correction coefficient and the contrast correction coefficient control. Based on the amount αi, the color information vCi is corrected, and the corrected color information is output to the output unit 13 as corrected color information vCi_new.
<6.2: Operation of Image Processing Device>
FIG. 30 shows a process flowchart of an image processing method (image processing method in the image processing apparatus 600) according to the sixth embodiment. The image processing apparatus 600 is an apparatus that corrects color information in an image by processing image data. The image processing apparatus 600 includes, for example, photographing devices such as digital still cameras and digital video cameras, image editing devices that edit digital images acquired by these photographing devices, mobile phones and car mobile devices used in a mobile environment, It is mounted on a PDA or the like, or a large video display device used in various environments.

A feature of the image processing apparatus 600 and the image processing method according to the sixth embodiment is that, in the first embodiment, the correction coefficient control amount α for determining the correction amount by the color contrast, which is a fixed number, is the color contrast amount RCi, the hue. Control is performed using Hi, brightness information Yi, brightness contrast amount RYi, and correction target color information Ci. That is, the sixth embodiment is obtained by adding the features of the second to fifth embodiments to the first embodiment. Therefore, as is apparent from the processing flowchart shown in FIG. 30 , image data is input to the image processing apparatus 600, and data of each pixel is converted into predetermined color information vCi. At the same time, the brightness information calculation unit 50 calculates the brightness information Yi, and the hue information calculation unit 40 calculates the hue information Hi.
Next, the brightness characteristic calculation unit 60 calculates the brightness contrast amount RYi, and the color characteristic calculation unit 20 calculates the color contrast amount RCi. In response to these results, the contrast correction amount control unit 6030 controls the correction coefficient control amount α. That is, the contrast correction amount control unit 6030 calculates the correction coefficient control amount α based on the brightness information Yi, the brightness contrast amount RYi, and the hue information Hi. Then, by changing the correction coefficient control amount α, the image processing apparatus 600 can generate a more desired image.

The output unit 13 may output the correction value of the target color information Ci of each pixel Pi obtained by the extended color information correction unit 80 and the other color information values as output image data as they are. In addition, the correction value of the target color information Ci (for example, saturation) and the other color information values (for example, hue and brightness) of each pixel Pi obtained by the extended color information correction unit 80 are handled by the device used. It may be converted into an image format that can be output and output as output image data.
The correction coefficient control amount α can be implemented by combining the controls in the second to fifth embodiments, for example. Further, in the space constituted by the above-described parameters and the correction coefficient control amount α, an area belonging to the space may be divided and an appropriate correction coefficient control amount α may be defined in each area. In this case, in order to make the vicinity of the boundary of each area smooth, the boundary of each area may be connected to improve a color correction defect that may occur in the vicinity of the boundary.

Further, the correction coefficient control amount α is defined as a smooth function that is regarded as a function of the color contrast amount RCi, hue Hi, brightness information Yi, brightness contrast amount RYi, and correction target color information Ci. It may be.
In this way, the correction amount based on the color contrast can be controlled more flexibly and in detail. That is, by performing processing with the image processing apparatus 600, it is possible to create an image with improved feeling of depth more flexibly. Further, the image processing apparatus 600 does not cause a large deterioration in image quality or impression due to a decrease in resolution, which is a problem in the prior art.
In the above description, the processing of the present invention in the case where the target color information Ci is one has been described. However, for example, color information correction may be executed by combining two color information corrections of saturation and hue. Is possible. In this case, color characteristic information data from each color information is obtained separately, and the extended color information correction unit 80 sets a correction function suitable for the corresponding color information.

In the above description, the correction gain GCi based on the depth information fbi is calculated, and the method of correcting the correction amount based on the calculated correction gain GCi and the color contrast amount is described in the present invention. For example, the present invention may be realized by a method of executing color information correction by combining the color correction amount based on the depth information fbi and the color correction amount based on the color contrast.
[Seventh Embodiment]
31 to 36 , as a seventh embodiment of the present invention, an image processing method for estimating depth information based on color contrast information in a pixel, and correcting color information according to the result, and The image processing apparatus 700 will be described.
FIG. 31 shows the configuration of an image processing apparatus 700 according to the seventh embodiment of the present invention. FIG. 32 shows the configuration of the depth estimation unit 1006 of the image processing apparatus 700.

FIG. 33 shows a processing flowchart of the image processing method according to the seventh embodiment, and FIG. 34 shows a processing flowchart of the depth estimation step in the image processing method according to the seventh embodiment.
An image processing apparatus 700 according to the seventh embodiment mainly includes a color information calculation unit 1004, a depth estimation unit 1006, a depth correction unit 1008, and an output unit 1010. The depth estimation unit 1006 includes a color characteristic calculation unit 1020 and a depth degree calculation unit 1023. The color characteristic calculation unit 1020 includes a peripheral representative color calculation unit 1021 and a color contrast amount calculation unit 1022.
The image processing apparatus 700 is an apparatus that estimates depth information in an image and corrects the depth by processing image data. The image processing device 700 is, for example, a photographing device such as a digital still camera or a digital video camera, an image editing device that edits a digital image acquired by these photographing devices, a mobile phone or a car mobile device used in a mobile environment, It is mounted on a PDA or the like, or a large video display device used in various environments.

The image processing method and the image processing apparatus 700 according to the seventh embodiment of the present invention will be described with reference to FIGS . 33 and 34 .
First, when image data is input to the image processing apparatus 700, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Here, the image input data includes HSV space data composed of hue H, saturation S, and lightness V that can easily handle color information, YCbCr space data composed of luminance Y, color difference Cb, and Cr, lightness L, and color. The image data is converted into La * b * space data composed of a * and b *, but the image input data can be handled as it is.
The depth estimation unit 1006 receives the color information vCi and estimates depth information in the image. At this time, first, the color characteristic calculation unit 1020 calculates characteristic information for the color information Ci to be corrected among the color information vCi obtained by the color information calculation unit 1004. For example, when the color is converted to the HSV system and the saturation S is corrected, the characteristic information regarding the saturation S is calculated. Here, the color information to be corrected among the color information vCi in the pixel Pi is Ci.

As shown in FIG. 32 , the color characteristic calculation unit 1020 generates color characteristic information data for each pixel through the peripheral representative color calculation unit 1021 and the color contrast amount calculation unit 1022. Many color characteristic information data i can be considered. In order to express a visual characteristic closer to a human appearance, a color contrast amount RCi corresponding to the human visual characteristic is used. There are many human visual characteristics, and in addition to color contrast amounts, brightness contrast amounts, size / pixel characteristics, and the like can also be used.
The color characteristic calculation unit 1020 calculates color contrast amount data RCi for each pixel.
First, the peripheral representative color calculation unit 1021 calculates the representative color information ACi around the target pixel Pi. In this case, an area of about 1/6 to 1/3 of the image (all image areas) is set as a peripheral area (viewing area), and weighted average color information ACi is obtained.

The color contrast amount calculation unit 1022 obtains the color contrast amount RCi of the pixel Pi. Here, the ratio of the target color information Ci of the target pixel Pi to the representative color information ACi is defined as a color contrast amount RCi. Besides this, it is possible to define the color contrast information data RCi as (Ci-ACi).
Next, the depth degree calculation unit 1023 estimates depth information in the image using the color contrast amount RCi. FIG. 35 schematically shows the concept. In FIG. 35 , the contrast based on the saturation is described as the color contrast, but other color relative ratios and the like can be considered in the same manner qualitatively. There are two petals in FIG. 35. The petals of the two petals have the same color information saturation. In contrast, assume that the center of the left petal is more saturated than the surrounding petals, and the center of the right petal is less saturated than the surrounding petals. In this case, when the saturation contrast amount RSPc obtained at the pixel Pc in the left petal center and the saturation contrast amount RSQc obtained at the pixel Qc in the right petal center are compared, RSP c > RSQc. As a result, due to the influence of the saturation contrast, the center of the left petal feels more vivid than the actual saturation SPc in terms of visual characteristics, and the center of the right petal is lighter than the actual saturation SQc. It will be felt. Therefore, it can be determined that the center of the left petal is likely to be included in the foreground because the human attention level is higher. On the other hand, in the right petal center part, it can be determined that the possibility of being included in the foreground is low due to the reduction of human attention.

In the present invention, based on the facts described above, a pixel having a high color contrast amount RCi of the target pixel is regarded as a pixel included in a region of high attention, that is, a pixel included in a foreground region, and the depth degree of the pixel. Set a high value for fbi. The depth degree fbi satisfies 0.0 ≦ fbi ≦ 1.0, and fbi = 1.0 indicates the highest possibility of a foreground, and fbi = 0.0 indicates the lowest possibility of a foreground ( Distant view). FIG. 36 shows the relationship between the color contrast amount RCi and the depth degree fbi. A pixel having a color contrast amount RCi of RCi = 1.0 is regarded as being at an intermediate distance between the foreground and the background, and the depth degree fbi is set to “0.5”. As the color contrast amount RCi increases toward the maximum color contrast amount MaxRCi, the possibility of belonging to the foreground increases, and the degree of depth fbi also monotonously increases toward “1.0”. Conversely, as the color contrast amount RCi decreases toward “0.0”, the depth degree fbi is also monotonously decreased toward “0.0”, assuming that the possibility of belonging to the foreground is reduced. After defining the depth fbi of each pixel in this way, this value is used as a correction gain when performing color correction of the correction target color information Ci executed by the depth correction unit 1008. As the color correction, for example, the correction color information Ci_new may be obtained by changing as shown in (Equation 17 ) based on a predetermined fluctuation amount DefCi.

Here, Ci_new is a value after correction of the correction target color information. β is a predetermined positive constant.
In addition, it is possible to perform color correction by color contrast. In this case, as an example, it is also possible to obtain the correction color information Ci_new by (Equation 18 ) and (Equation 19 ).

TH0, Gmax, and β are predetermined positive constants, and μ is a predetermined positive constant that satisfies 0 <μ <1. (Formula 18 ) is generated by abrupt changes in the color information Ci to be corrected by adding the correction target Ci term to the correction amount by color contrast (by multiplying the two items of (Formula 18 ) by Ci). It is an example of the formula which calculates | requires correction | amendment color information Ci_new so that the saturation phenomenon on a process image may be suppressed. Note that color correction is an example for improving the depth shown here, and many methods are conceivable.
Finally, the output unit 1010 may output the correction value of the target color information Ci of each pixel Pi obtained by the depth correction unit 1008 and other color information values as output image data as they are. The correction value of the target color information Ci (for example, saturation) of each pixel Pi obtained by the depth correction unit 1008 and other color information values (for example, hue and brightness) can be handled by the device used. It may be converted into a possible image format and output as output image data.

In the image processing apparatus 700, by estimating the degree of depth using the color contrast amount in this way, a weak contour portion such as a texture pattern or a contour portion that cannot be appropriately extracted due to shooting conditions such as ambient light. However, depth estimation is possible. Furthermore, in the image processing apparatus 700, it is not necessary to perform further threshold processing on the secondary differential signal of the pixel considered to be a contour portion as in the prior art, and therefore the influence of the threshold determination accuracy also in depth information determination. Not receive.
Further, even if there is a blur area that does not affect the depth sensation that has been a problem when using a second-order differential signal as in the prior art (not related to the depth sensation), the color information of the target portion is displayed from the surroundings. Since it is possible to obtain a color contrast indicating whether the color is large, it is possible to extract a portion having a high color contrast. It has also been pointed out that humans tend to focus more attention on areas with high color contrast, and in the present invention, depth information in an image is simply estimated by regarding a region with high color contrast as a foreground. be able to.

In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.
[Eighth Embodiment]
37 to 39 , as an eighth embodiment of the present invention, an image processing method for estimating depth information based on color contrast information in a pixel and correcting color information according to the result, and An image processing apparatus will be described.

FIG. 37 shows the configuration of the depth estimation unit 806 in the image processing apparatus according to the eighth embodiment of the present invention. FIG. 38 shows a flowchart of the depth estimation process of the image processing method according to the eighth embodiment.
The image processing apparatus according to the eighth embodiment is obtained by replacing the depth estimation unit 1006 with a depth estimation unit 806 in the image processing apparatus 700 according to the seventh embodiment. This is the only difference between the image processing apparatus according to the eighth embodiment and the image processing apparatus 700 according to the seventh embodiment, and the other parts are the same. Description is omitted.
The depth estimation unit 806 in the image processing apparatus according to the eighth embodiment mainly includes a color characteristic calculation unit 1020, a foreground pixel determination unit 1030, a foreground position estimation unit 1031 and a gain calculation unit 1032. The same parts as those of the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.

Figure 31, in accordance with FIG. 37, described eighth image processing apparatus and image processing method according to an embodiment of the present invention.
When image data having a pixel value vIi at a pixel i is input to the image processing apparatus of this embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. The Here, the image input data includes HSV space data composed of hue H, saturation S, and lightness V that can easily handle color information, YCbCr space data composed of luminance Y, color difference Cb, and Cr, lightness L, and color. The image data is converted into La * b * space data composed of a * and b *, but the image input data can be handled as it is. The peripheral representative color calculation unit 1021 obtains the representative color information ACi of the target color information Ci in the peripheral region of the pixel i, and the color contrast amount calculation unit 1022 obtains the color contrast amount RCi as in the above-described embodiment. It is the same.

Then, the foreground pixel determination unit 1030 determines a pixel group Gr composed of foreground pixel candidates. The left figure of FIG. 39 is a figure for demonstrating this candidate selection process typically. There are many methods for determining the pixel group Gr. For example, a method may be employed in which a predetermined threshold ThRC is set and a pixel i having an RCi larger than that value is included in Gr. Further, as shown in the left diagram of FIG. 39, a method may be employed in which a threshold ThRC is determined from a histogram of the color contrast amount RCi in the image, and pixels having RCi larger than the threshold are included in Gr. Further, an average value ARC of the color contrast amount RCi in the image and its standard deviation dRC are obtained,
Gr = {i | ARC−dRC ≦ RCi ≦ ARC + dRC, 0 ≦ i ≦ NNN−1}
It is also possible to adopt the method obtained as

Here, it is sought foreground pixel candidates Gr left diagram of the method of Figure 39.
Next, the foreground position estimation unit 1031 obtains the foreground position vCen (CenX, CenY) using this Gr. As shown in FIG. 39 , when the number of pixels belonging to Gr is NGr, the foreground position vCen (CenX, CenY) is obtained as in (Equation 20 ). In addition to this, it is also possible to adopt a method of determining the foreground position vCen (CenX, CenY) from the pixel position having the largest color contrast amount in Gr and the histogram for the color contrast amount in Gr. . In (Expression 20 ), (Xk, Yk) indicates (X coordinate, Y coordinate) of the pixel k.

This (CenX, CenY) corresponds to an average coordinate of what is likely to be included in the foreground. The foreground pixel determination process includes threshold processing, but the representative value and average value of a plurality of pixels determined to be highly likely to be included in the foreground pixel in the foreground pixel determination process are used as the foreground position vCen (CenX, CenY). By doing so, the foreground position estimation error due to fluctuations in the image can be suppressed to some extent.
As shown in the lower right diagram of FIG. 39 , the gain calculation unit 1032 applies a convex function centered on the foreground position vCen (CenX, CenY) to the pixel i (X, Y) to obtain the pixel A correction gain value Gi at i is obtained. As shown in the lower right diagram of FIG. 39 , by approximating the correction gain Gi with a continuous function, it is possible to avoid a sudden change in the correction gain Gi, and to reduce the adverse effects on the corrected image. In addition, since the foreground position vCen can be regarded as a region with high attention, it can be determined that the vicinity of the region with high attention is close to some extent. On the other hand, as the distance from vCen increases, the degree of attention decreases and the possibility of going to a distant view increases. Therefore, the correction gain Gi is determined by a convex function as shown in FIG. 39 or (Equation 21 ). As long as the function satisfies this characteristic, the function for determining the correction gain Gi is not limited to (Equation 21 ), and may be another linear function or a non-linear function. It may be a quadric surface function.

Here, len indicates a square distance between the pixel i (X, Y) and the foreground position vCen (CenX, CenY), and ThDelta is a predetermined positive constant and indicates the extent of the convex function. is there.
In response to the correction gain Gi determined in this way, according to the present invention, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with an improved sense of depth is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
In the present invention, by performing the above-described processing, a weak contour portion such as a texture pattern and a contour portion that could not be extracted well due to photographing conditions such as ambient light as in the conventional example, and blur Even if there is an image area, depth estimation is sufficiently possible. In addition, the foreground candidate selection process includes a threshold process. By using the representative coordinates and average coordinates of a plurality of selected candidates as the foreground position, the influence of threshold determination accuracy can be suppressed in the present invention. it can.

Furthermore, in the present invention, a pixel having a large color contrast amount is selected as a foreground candidate pixel as having a high degree of attention and likely to be included in the foreground region, and the foreground position is estimated from the selected foreground candidate pixel. Thus, even when the influence of fluctuation due to noise or the like in the image, which is a problem in the case of the seventh embodiment, is present in the color contrast amount, the foreground area can be estimated with a certain degree of accuracy. Accordingly, in the present invention, it is possible to appropriately realize the depth sensation estimation according to the degree of human interest and the color correction based on the depth sensation estimation. Therefore, according to the present invention, an image with improved depth sensation can be obtained. Can be acquired.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

[Ninth Embodiment]
40 to 42 , as a ninth embodiment of the present invention, an image processing method for estimating depth information based on color contrast information in a pixel and correcting the color information according to the result, and An image processing apparatus will be described.
FIG. 40 shows the configuration of the depth estimation unit 906 in the image processing apparatus according to the ninth embodiment of the present invention. FIG. 41 shows a flowchart of depth estimation processing of the image processing method according to the ninth embodiment.
The image processing apparatus according to the ninth embodiment is obtained by replacing the depth estimation unit 1006 with a depth estimation unit 906 in the image processing apparatus 700 according to the seventh embodiment. This is the only difference between the image processing apparatus according to the ninth embodiment and the image processing apparatus 700 according to the seventh embodiment, and the other parts are the same. Description is omitted.

The depth estimation unit 906 in the processing apparatus according to the ninth embodiment mainly includes a block division unit 1040, an in-block color average unit 1041, a block color characteristic calculation unit 1042, a foreground block determination unit 1043, and a foreground position. An estimation unit 1031 and a gain calculation unit 1032 are included. The block color characteristic calculation unit 1042 mainly includes a block peripheral representative color calculation unit 1044 and a block color contrast amount calculation unit 1045. In addition, the same code | symbol is attached | subjected about the part similar to the above-mentioned embodiment, and description is abbreviate | omitted.
When image data having a pixel value vIi at a pixel i is input to the image processing apparatus of this embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. The Here, the image input data includes HSV space data composed of hue H, saturation S, and lightness V that can easily handle color information, YCbCr space data composed of luminance Y, color difference Cb, and Cr, lightness L, and color. The image data is converted into La * b * space data composed of a * and b *, but the image input data can be handled as it is.

The block dividing unit 1040 divides the image to be processed into blocks of a predetermined fixed size, and the intra-block color averaging unit 1041 is the color information Ci that is the correction target in each block k subdivided by the block dividing unit 1040. The average value ACk is obtained.
The block color characteristic calculation unit 1042 is different from the color characteristic calculation unit 1020 in the seventh embodiment in which the color contrast amount RCi is obtained in units of pixels, whereas the color contrast amount BRCk in units of blocks (k is the target block number). Ask for. Therefore, the block peripheral representative color calculation unit 1044 may be a representative value (a weighted average value or a value calculated from a histogram) of the average color information ACm in the block m in the peripheral region of the target block k. ) Find ABCk.
The block color contrast amount calculation unit 1045 obtains a block color contrast amount RBCk defined by a ratio, a difference, or the like of the average color information ACk of the target block k to ABCk. These processes are performed for all blocks k.

Then, the foreground block determination unit 1043 performs foreground candidate processing performed by the foreground pixel determination unit 1030 in the eighth embodiment on a block basis. Here, assuming that the selected foreground candidate block set is GBr, the foreground position estimation unit 1031 calculates the foreground position vCen (CenX, CenY) from the average value of the barycentric coordinates (XBk, YBk) of the selected foreground candidate block k. Ask for.
Thereafter, as in the eighth embodiment, based on the square distance len between the pixel i (X, Y) in the image and the foreground position vCen (CenX, CenY), The gain calculation unit 1032 performs a process of obtaining the correction gain value Gi of the pixel i by using it.
In response to the correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.

Finally, the output unit 1010 performs the same processing as in the previous embodiment.
As described above, the image processing apparatus according to the ninth embodiment is characterized in that depth estimation performed in units of pixels is performed in units of blocks, and an effect thereof is an increase in processing speed. Further, in the image processing apparatus of the present embodiment, the same effect as performing the low-pass filter process on the color information Ci by changing the color information Ci of the pixel to the average value ACk in the block k to which the pixel belongs. Can be realized. Thereby, in the image processing apparatus according to the present embodiment, fluctuation due to illumination conditions and color cast due to partial illumination can be suppressed, and depth estimation accuracy can be improved.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
[Tenth embodiment]
43 to 45 , as a tenth embodiment of the present invention, an image processing method for estimating depth information based on color contrast information in a pixel and correcting the color information according to the result, and An image processing apparatus will be described.

FIG. 43 shows the configuration of the depth estimation unit 10006 in the image processing apparatus according to the tenth embodiment of the present invention. FIG. 44 shows a flowchart of depth estimation processing of the image processing method according to the tenth embodiment.
The image processing device according to the tenth embodiment is obtained by replacing the depth estimation unit 1006 with a depth estimation unit 10006 in the image processing device 700 according to the seventh embodiment. This is the only difference between the image processing apparatus according to the tenth embodiment and the image processing apparatus 700 according to the seventh embodiment, and the other parts are the same. Description is omitted.
The depth estimation unit 10006 in the image processing apparatus of the tenth embodiment mainly includes a block division unit 1040, an intra-block color average unit 1041, a block color characteristic calculation unit 1042, and an intra-block gain calculation unit 1050, and a synthesis. And a gain calculation unit 1051. In addition, the same code | symbol is attached | subjected about the part similar to the above-mentioned embodiment, and description is abbreviate | omitted.

When image data having a pixel value vIi at a pixel i is input to the image processing apparatus of this embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. The Here, the image input data includes HSV space data composed of hue H, saturation S, and lightness V that can easily handle color information, YCbCr space data composed of luminance Y, color difference Cb, and Cr, lightness L, and color. The image data is converted into La * b * space data composed of a * and b *, but the image input data can be handled as it is.
The block dividing unit 1040 divides the image to be processed into blocks of a predetermined fixed size, and the intra-block color averaging unit 1041 is the color information Ci that is the correction target in each block k subdivided by the block dividing unit 1040. The average value ACk is obtained.
The block color characteristic calculation unit 1042 obtains a color contrast amount BRCk (k is a target block number) in units of blocks. The processing so far is the same as that of the above-mentioned embodiment.

The intra-block gain calculation unit 1050 performs the following processing.
(I) First, the intra-block gain calculation unit 1050 sets MaxRCk obtained from the color contrast amount BRCk of the block k as a correction gain for the center of gravity vCenk (CenXk, CenYk) of the block k. Here, MaxRCk is calculated using a conversion function MaxRCk = FFunc (BRCk) that outputs a value in the range of “0.0” to “1.0”. This conversion function MaxRCk = FFunc (BRCk) is defined by a monotonic linear increase function in which the output at the maximum value MaxRC that BRCk can take is “1” and the output at BRCk = 0 is “0”. Note that this function is not limited to a linear function, and may be a nonlinear function. Further, this function is compared with a predetermined maximum color contrast amount MaxRC. When BRCk is equal to or less than a predetermined ratio CRate with respect to MaxRC, MaxRCk = BRCk, and BRCk is based on a predetermined ratio CRate with MaxRC. If larger, MaxRCk = 1.0 can also be set.
(Ii) In the intra-block gain calculation unit 1050, a correction gain value Gk_i for the pixel i (X, Y) based on the color contrast amount BRCk of the block k is defined as (Equation 22 ). That is, the function for obtaining the correction gain value Gk_i is defined by a convex function having a maximum value MaxRCk at vCenk (CenXk, CenYk) and changing according to the square distance length between vCenk and i. This function indicates that the possibility of the near view by the color contrast amount BRCk decreases as it goes to the distant view with vCenk as the center. It should be noted that ThDelta in (Expression 22 ) indicates the spread of this function, and is constant regardless of the block here, but may be changed according to the size of the block or BRCk.

The intra-block gain calculation unit 1050 performs the above processing.
(Iii) The sum of the correction gains Gk_i for the pixel i by all the blocks is obtained, and a value obtained by normalizing the obtained value from “0” to “1” is set as the correction gain Gi.

The combined gain calculation unit 1051 performs processing corresponding to this (Equation 23 ).
FIG. 45 is an explanatory diagram schematically showing the above.
As shown in FIG. 45 , in the image processing apparatus according to the present embodiment, the correction gain Gk_i representing the possibility of the near view of the pixel i in the case of the color contrast amount BRCk by the block k is considered in consideration of the influence of all the blocks. Finally, the correction gain Gi of the pixel i is obtained.
In the case of the ninth embodiment, fluctuations in the color contrast amount RCi due to fluctuations and fluctuations in the color information Ci in the image may cause fluctuations in the foreground position estimation. In the case of the ninth embodiment, in particular, in the case of a moving image, the color contrast amount RCi of the same pixel i fluctuates due to problems such as lighting conditions and encoding even if the scene is a continuous frame and the scene does not change. There is. In this case, depending on the threshold ThRC at the time of selecting a foreground pixel or a foreground block candidate, there is a possibility that a shift of the foreground position vCen may occur, and the processing is performed by a function approximated by a specific convex function centered on the foreground position vCen. When doing so, there is a risk of image flickering. However, in the image processing apparatus of this embodiment, the sub correction gain Gk_i based on the color contrast amount BRCk of the block is set at the center of each block without obtaining the specific foreground position vCen, and is determined by all blocks. By considering the sub correction gain Gk_i, fluctuations in the correction gain Gi necessary for depth correction can be suppressed.

In response to the correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
As described above, by using the processing in the tenth embodiment, it is possible to expect high-speed processing by performing the processing in units of blocks. In the image processing apparatus according to the present embodiment, the above-described processing may cause a flicker of a corrected image, particularly a frame image that is a continuous frame of a moving image and has no scene change, and may occur in the case of the ninth embodiment. In addition, it is possible to more effectively suppress the minute fluctuation of the near view position and the fluctuation of the correction gain caused by the fluctuation.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
[Eleventh embodiment]
46 to 52 , as eleventh embodiment of the present invention, depth information is estimated based on color contrast information and brightness contrast information in a pixel, and color information is corrected according to the result. An image processing method to be performed and the image processing apparatus 1100 will be described.

FIG. 46 shows the configuration of an image processing apparatus 1100 according to the eleventh embodiment of the present invention.
The image processing apparatus 1100 mainly includes a brightness information calculation unit 1101, a color information calculation unit 1004, a second depth estimation unit 1103, a depth correction unit 1008, and an output unit 1010.
FIG. 47 shows a configuration of the second depth estimation unit 1103 in the image processing apparatus 1100.
The second depth estimation unit 1103 mainly includes a color characteristic calculation unit 1020, a brightness characteristic calculation unit 1110, and an integrated depth degree calculation unit 1113.
The brightness characteristic calculation unit 1110 includes a peripheral representative brightness calculation unit 1111 and a brightness contrast amount calculation unit 1112.
In addition, the same code | symbol is attached | subjected about the part similar to the above-mentioned embodiment, and description is abbreviate | omitted.

The second depth estimation unit 1103 receives the brightness information Yi calculated by the brightness information calculation unit 1101 and the color information vCi calculated by the color information calculation unit 1004, and is based on the brightness information Yi and the color information vCi. The correction gain Gi is calculated, and the calculated correction gain Gi is output to the depth correction unit 1008.
FIG. 48 shows an overall process flowchart of the present invention. FIG. 49 shows a processing flowchart of the depth estimation step. Further, FIG. 50 shows a processing flow of the color characteristic calculation step and a processing flow of the brightness characteristic calculation step. The processing of the color characteristic calculation step and the color characteristic calculation unit are the same as those in the seventh to tenth embodiments.
The feature of the invention of this embodiment is that, in the seventh embodiment, the depth degree is calculated from the color characteristic amount RCi which is the color characteristic information. The brightness characteristic (brightness contrast amount) is one of the visual characteristics. RYi) is further used to calculate the depth degree, thereby improving the depth estimation accuracy.

First, when image data having the pixel value vIi at the pixel i is input to the image processing apparatus of the present embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. The conversion is the same as in the seventh embodiment.
Next, the brightness information calculation unit 1101 obtains brightness information Yi from vIi. There are many examples of brightness information, such as brightness Y of YCbCr space data composed of brightness Y, color difference Cb, Cr, and La * b * space data composed of brightness L, color a *, b *. The lightness L is one of them. Here, the luminance Y in the YCbCr space data is calculated by the pixel i and used as the brightness information Yi.
The second depth estimation unit 1103 receives the color information vCi and the brightness information Yi and starts processing. First, the color characteristic calculation unit 1020 obtains the color contrast amount of the color information Ci to be corrected / estimated from vCi in the same manner as in the seventh embodiment. On the other hand, for the brightness information Yi, the brightness characteristic calculation unit 1110 obtains information indicating the brightness characteristic according to the visual characteristic. Many things can be considered to indicate this brightness characteristic, but in order to realize a correction that is closer to the human appearance, information corresponding to the human visual characteristic is used. There are many human visual characteristics, and brightness contrast characteristics are used.

FIG. 51 is a schematic diagram for explaining brightness contrast characteristics.
In the image processing apparatus 1100 of the present embodiment, the brightness contrast amount is used in consideration of the brightness contrast characteristics related to the brightness characteristics. It is known that the color contrast with respect to the color also occurs in the brightness information, and the brightness contrast amount is a numerical value of the degree. For example, consider an example in which there is a small center circle with higher brightness than the surrounding area at the center of a large circle with low brightness. In this case, it is clear from visual psychology that the center of a large circle tends to feel brighter than actual brightness. This phenomenon is a phenomenon caused by brightness contrast characteristics, and occurs when the brightness of an object is affected by the brightness of the surroundings when different brightness surrounds the target object.
That is, when the brightness surrounding the object is higher than the brightness of the object itself, the brightness of the object is felt low. Conversely, when the brightness surrounding the object is lower than the brightness of the object itself, the brightness of the object is felt high.

  In the present invention, this brightness contrast amount is combined with the color contrast amount and used for depth estimation, thereby obtaining depth information that humans feel more accurately. In addition, as a peripheral region when obtaining the brightness contrast amount RYi, processing is performed using a pixel range (region) that occupies about 1/6 to 1/3 of an image corresponding to a human visual field, as in the case of color contrast. It was decided to. At this time, as the representative brightness AYi of the pixels around the target pixel Pi, for example, the weighted average brightness in the pixel area Ωi having a predetermined area corresponding to the human visual field area is set as the surroundings of the target pixel Pi. It is preferable to use the representative brightness AYi of the pixel of the pixel, but in addition to this, for the pixel in the field of view, a histogram of the brightness (luminance) of the pixel in the region is obtained. The representative brightness AYi can be the brightness value with the highest frequency, the representative brightness obtained by clustering processing based on the statistical distribution in the field of view, the average brightness in the field of view, etc. is there.

In addition, as a definition of the brightness contrast amount RYi,
(1) The ratio of the brightness information Yi to be corrected to the brightness information AYi representing the periphery thereof,
(2) A value obtained by subtracting brightness information AYi representing the periphery from brightness information Yi to be corrected,
Many definitions can be adopted. Note that many other definitions are possible as in the case of the color contrast amount RCi.
The surrounding representative brightness calculation unit 1111 obtains the surrounding representative brightness AYi, and the brightness contrast amount calculation unit 1112 obtains the brightness contrast amount RYi.
The integrated depth degree calculation unit 1113 obtains the depth degree fbi of the target pixel using the color contrast amount RCi and the brightness contrast amount RYi. FIG. 52 is an explanatory diagram of an example of the method. As shown in (Equation 24 ) and FIG. 52 , the integrated depth degree calculation unit 1113 obtains a depth degree fb1_i based on color contrast and a depth degree fb2_i based on brightness contrast, and calculates the depth of the pixel i by taking these two products. The degree fbi is calculated.

The depth degree fb1_i here is the same as in the above-described embodiment. A pixel having a high color contrast amount RCi of the target pixel is regarded as a pixel included in a region of high attention, that is, a pixel included in a foreground region. A high value is set for the pixel depth degree fb_i. Note that the depth degree fb1_i takes a value between 0.0 ≦ fb1_i ≦ 1.0, and fb1_i = 1.0 has the highest possibility of near view, and fb1_i = 0.0 has the lowest possibility of close view ( It is defined as (distant view).
Also, as shown in the lower right diagram of FIG. 52, a pixel having a high brightness contrast amount RYi is likely to be included in a region with a high degree of attention, and is considered to be included in a foreground region, and thus the brightness contrast amount RYi. The depth degree fb2_i is set to a high value. Similar to the depth degree fb1_i based on the color contrast RCi, the depth degree fb2_i takes a value between 0.0 ≦ fb2_i ≦ 1.0, and fb2_i = 1.0 has the highest possibility of near view, and fb2_i = 0. 0 is defined as the lowest possibility of near view (far view). When the brightness contrast amount RYi is RYi = 1.0, it is assumed that the brightness contrast amount RYi is at an intermediate distance between the foreground and the background, the depth degree fb2_i is set to “0.5”, and the brightness contrast amount RYi is the maximum brightness. As the possibility of belonging to the foreground increases as the contrast amount MaxRYi increases, the degree of depth fb2_i also monotonously increases toward “1.0”. Conversely, as RCi decreases toward “0.0”, the possibility of belonging to the foreground decreases, and the degree of depth fb2_i also decreases monotonously toward “0.0”. Based on the above, the depth degree fb2_i based on the brightness contrast of each pixel is defined (determined).

As shown in (Equation 24 ), after the depth degree fbi is calculated, the calculated depth degree fbi is set as a correction gain Gi for color correction of the correction target color information Ci. In other words,
(Correction gain Gi) = fb1_i × fb2_i
And
The depth degree fbi of the pixel i, that is, the correction gain Gi is not limited to the example here, and a value approximated by a function having the color contrast amount RCi and the brightness contrast amount RYi as variables is used. Is also possible. In this case, the maximum correction gain “1.0” is set at a position where RCi and RYi are the highest, and the value of the correction gain Gi is set by a function (such as a linear function or a non-linear function) that gently attenuates around that position. Setting (calculating) can be considered. Also, in a two-dimensional space defined by RCi and RYi, a region in the two-dimensional space is divided, each region is set to a constant correction gain, and smoothly continuous at the boundary dividing the region. The value of the correction gain Gi may be set (calculated) using a function having the following characteristics.

In response to the correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
In the present invention, by performing such processing, it is determined that there is a high possibility that a portion with a high color contrast is a foreground, as in the seventh embodiment, and the depth degree is also considered in consideration of the brightness contrast. Therefore, a region that is more strongly felt by humans can be properly regarded as a foreground region, and the calculation accuracy of the correction gain Gi for performing depth estimation, that is, depth correction can be further improved.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

[Twelfth embodiment]
53 to 55 , as a twelfth embodiment of the present invention, depth information is estimated based on color contrast information and brightness contrast information in a pixel, and color information is corrected according to the result. An image processing method and an image processing apparatus to be performed will be described.
FIG. 53 shows the configuration of the second depth estimation unit 1203 in the image processing apparatus according to the twelfth embodiment of the present invention. FIG. 54 shows a flowchart of the depth estimation process of the image processing method according to the twelfth embodiment.
The image processing device according to the twelfth embodiment is obtained by replacing the second depth estimation unit 1103 with a second depth estimation unit 1203 in the image processing device 1100 according to the eleventh embodiment. This is the only difference between the image processing apparatus according to the twelfth embodiment and the image processing apparatus 1100 according to the eleventh embodiment, and the other parts are the same. Description is omitted.

The second depth estimation unit 1203 in the image processing apparatus of the twelfth embodiment mainly includes a color characteristic calculation unit 1020, a brightness characteristic calculation unit 1110, an integrated foreground pixel determination unit 1130, a foreground position estimation unit 1031, And a gain calculation unit 1032. The same parts as those of the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
The feature of the invention of the present embodiment is that in the eighth embodiment, whether or not the pixel is a foreground pixel is determined based on the color contrast amount RCi of the pixel i. The brightness contrast amount RYi and the color contrast amount RCi are The feature point is that it is determined whether or not a foreground pixel is present.
In the image processing apparatus according to the present embodiment, the processes by the color information calculation unit 1004, the brightness information calculation unit 1101, the color characteristic calculation unit 1020, and the brightness characteristic calculation unit 1110 are the same as those in the above-described embodiment, and thus will be described. Is omitted.

As shown in FIG. 55 , the integrated foreground pixel determination unit 1130 receives the color contrast amount RCi and the brightness contrast amount RYi of the pixel i, and determines whether the pixel i is a foreground pixel candidate. Then, a pixel group Gr composed of foreground pixel candidates is determined. The left figure of FIG. 55 is a figure for demonstrating this candidate selection typically. There are many methods for determining the pixel group Gr. For example, a method of setting predetermined threshold values ThRC and ThRY and including a pixel i having RCi and RYi larger than the threshold values in the pixel group Gr may be adopted. As shown in the left diagram of FIG. 55, the threshold ThRC is determined from the histogram of the color contrast amount RCi in the image, and the threshold ThRY is determined from the histogram of the brightness contrast amount RYi in the image. A method of including pixels having large RCi and large RYi in Gr may be employed. Further, an average value ARC of the color contrast amount RCi in the image and its standard deviation dRC are obtained, and an average value ARY of the brightness contrast amount RYi and its standard deviation dRY are obtained,
Gr = {i | ARC−dRC ≦ RCi ≦ ARC + dRC, ARY−dRY ≦ RYi ≦ ARY + dRY, 0 ≦ i ≦ NNN−1}
It is also possible to adopt a method for obtaining the pixel group Gr.

Here, it is assumed that seek foreground pixel candidates Gr in the process of the left side of FIG. 55.
Next, the foreground position estimation unit 1031 obtains the foreground position vCen (CenX, CenY) using this Gr. In FIG. 55 , when the number of pixels belonging to Gr is NGr, the foreground position vCen (CenX, CenY) is obtained as in (Equation 20 ). In addition to this, it is also possible to adopt a method for determining the foreground position vCen (CenX, CenY) from an image having the maximum color contrast amount in Gr or a histogram for the color contrast amount in Gr. This foreground position vCen (CenX, CenY) corresponds to the average coordinates of those that are likely to be included in the foreground.
The gain calculation unit 1032 obtains the correction gain Gi of the pixel i using a function as shown in the lower right diagram of FIG . This uses the same convex function as the function represented by (Formula 21 ). If the function has the same characteristics as those in the lower right diagram of FIG. 55, the function is not limited to the function of (Equation 21 ), and a correction gain is obtained using a linear function or an upwardly convex quadric surface function. Gi may be calculated and the processing according to the present invention may be performed.

In response to the correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
In the image processing apparatus according to the present embodiment, the foreground candidate is selected based on the two feature amounts as described above. Therefore, the portion with the higher degree of human attention can be appropriately and more accurately compared to the case of the eighth embodiment. Can be selected as a region candidate.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

[Thirteenth embodiment]
56 to 58 , as a thirteenth embodiment of the present invention, depth information is estimated based on color contrast information and brightness contrast information in a pixel, and color information is corrected according to the result. An image processing method and an image processing apparatus to be performed will be described.
FIG. 56 shows the configuration of the second depth estimation unit 1303 in the image processing apparatus according to the thirteenth embodiment of the present invention. FIG. 57 shows a flowchart of depth estimation processing of the image processing method according to the thirteenth embodiment.
The image processing device according to the thirteenth embodiment is obtained by replacing the second depth estimation unit 1103 with a second depth estimation unit 1303 in the image processing device 1100 according to the eleventh embodiment. This is the only difference between the image processing apparatus according to the thirteenth embodiment and the image processing apparatus 1100 according to the eleventh embodiment, and the other parts are the same. Description is omitted.

The second depth estimation unit 1303 in the image processing apparatus of the thirteenth embodiment mainly includes a block division unit 1040, an intra-block color average unit 1041, an intra-block brightness average unit 1160, and a block color characteristic calculation unit 1042. A block brightness characteristic calculation unit 1161, an integrated foreground block determination unit 1162, a foreground position estimation unit 1031, and a gain calculation unit 1032. The same parts as those of the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
As schematically shown in FIG. 58 , the feature of the present embodiment is that in the ninth embodiment, whether or not the pixel is a foreground pixel is determined by the block color contrast amount BRCk related to the average color obtained in block k. A point is a point in which it is determined whether or not it is a close-up scene based on two feature amounts of a block contrast amount BRYk and a block color contrast amount BRCk related to the average brightness obtained in the same block k.

An image processing method and an image processing apparatus according to a thirteenth embodiment of the present invention will be described.
First, when image data having the pixel value vIi at the pixel i is input to the image processing apparatus of the present embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Conversion processing and conversion processing to brightness information Yi are performed.
Next, as in the ninth embodiment, the block dividing unit 1040 divides the image to be processed into blocks of a predetermined fixed size, and the block internal color averaging unit 1041 is subdivided by the block dividing unit 1040. The average value ACk of the color information Ci to be corrected in the block k is obtained.
The in-block brightness average unit 1160 obtains an average value AYk of brightness information Yi in each subdivided block k.

The block color characteristic calculation unit 1042 receives the block inner color average value ACk of the block k from the block inner color average unit 1041, and the block color contrast amount BRCk represented by the ratio of the surrounding block and the average color BCk of the target block k. Ask for.
Also, the block brightness characteristic calculation unit 1161 performs the process performed in units of pixels in the brightness characteristic calculation unit 1110 in the eleventh and twelfth embodiments in units of blocks, and calculates the brightness contrast amount BRYk of the block k. Ask.
The integrated foreground block determination unit 1162 obtains a group GBr of foreground block candidates based on the block color contrast amount BRCk and the block brightness contrast amount BRYk. The integrated foreground block determination unit 1162 performs the above-described determination process by performing the processing performed in units of pixels in the integrated foreground pixel determination unit 1130 in the twelfth embodiment.

The foreground position estimation unit 1031 obtains the foreground position vCen (CenX, CenY) from the average value of the barycentric coordinates (XBk, YBk) of the selected foreground candidate block k. Thereafter, as in the eighth embodiment, based on the square distance len between the pixel i (X, Y) in the image and the foreground position vCen (CenX, CenY), using (Formula 6), The gain calculation unit 1032 performs processing for obtaining the correction gain value Gi of the pixel i.
In response to this correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
As described above, the image processing apparatus according to the present embodiment is characterized in that the depth estimation performed in units of pixels is performed in units of blocks, and an effect thereof is an increase in processing speed.

In the image processing apparatus according to the present embodiment, the color information Ci and the brightness information Yi of the pixel are changed to the average values ACk and AYk in the block k to which the pixel belongs, whereby the color information Ci and the brightness information Yi are changed. On the other hand, it is possible to achieve the same effect as when low-pass filter processing is performed. Thereby, in the image processing apparatus of this embodiment, depth estimation accuracy can be further improved by suppressing fluctuations due to illumination conditions and the like and color casts due to partial illumination.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
[Fourteenth embodiment]
59 to 61 , as a fourteenth embodiment of the present invention, depth information is estimated based on color contrast information and brightness contrast information in a pixel, and color information is corrected according to the result. An image processing method and an image processing apparatus to be performed will be described.

FIG. 59 shows the configuration of the second depth estimation unit 1403 in the image processing apparatus according to the fourteenth embodiment of the present invention. FIG. 60 shows a flowchart of depth estimation processing of the image processing method according to the fourteenth embodiment.
The image processing device according to the fourteenth embodiment is obtained by replacing the second depth estimation unit 1103 with a second depth estimation unit 1403 in the image processing device 1100 according to the eleventh embodiment. This is the only difference between the image processing apparatus according to the fourteenth embodiment and the image processing apparatus 1100 according to the eleventh embodiment, and the other parts are the same. Description is omitted.
The second depth estimation unit 1403 in the image processing apparatus according to the fourteenth embodiment mainly includes a block division unit 1040, an intra-block color average unit 1041, an intra-block brightness average unit 1160, and a block color characteristic calculation unit 1042. A block brightness characteristic calculation unit 1161, an individual block gain calculation unit 1180, and an advanced combined gain calculation unit 1181.

The same parts as those of the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
In the image processing apparatus of the present embodiment, the value obtained from the block color contrast amount BRCi relating to the average color obtained in the block k is used as the correction gain Gk_i at the barycentric position vCenk (CenXk, CenYk) of the block k. The correction gain Gk_i based on the color contrast amount BRCk of the block k is obtained, and the sum of the correction gains Gk_i of all the blocks is finally set to the correction gain Gi of the pixel i . FIG. 61 shows the processing in the tenth embodiment . As described above, the correction gain Gi is set not only in the block color contrast amount BRCk but also in the block brightness contrast amount BRYk. This is a feature of the present embodiment.
An image processing method and an image processing apparatus according to the fourteenth embodiment of the present invention will be described.
First, when image data having the pixel value vIi at the pixel i is input to the image processing apparatus of the present embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Conversion processing and conversion processing to brightness information Yi are performed.

Next, as in the ninth embodiment, the block dividing unit 1040 divides the image to be processed into blocks of a predetermined fixed size, and the block internal color averaging unit 1041 is subdivided by the block dividing unit 1040. The average value ACk of the color information Ci to be corrected in the block k is obtained.
The in-block brightness average unit 1160 obtains an average value AYk of brightness information Yi in each subdivided block k.
The block color characteristic calculation unit 1042 receives the block inner color average value ACk of the block k from the block inner color average unit 1041, and the block color contrast amount BRCk represented by the ratio of the surrounding block and the average color BCk of the target block k. Ask for.
Further, the block brightness characteristic calculation unit 1161 obtains the brightness contrast amount BRYk of the block k. The above processing is the same as in the thirteenth embodiment.

The individual block gain calculation unit 1180 performs processing as an extended version of the block internal gain calculation unit 1050 in the tenth embodiment. First, the individual block gain calculation unit 1180 performs the following processing on the block color contrast amount BRCk of the block k.
(Ci) The individual block gain calculation unit 1180 sets MaxRCk obtained from the color contrast amount BRCk of the block k as the correction gain of the center of gravity vCenk (CenXk, CenYk) of the block k. Here, MaxRCk is calculated using a conversion function MaxRCk = FFunc (BRCk) that outputs a value in the range of “0.0” to “1.0”. The function MaxRCk = FFunc (BRCk) is defined as a monotonic linear increase function in which the output at the maximum value MaxRC that BRCk can take is “1” and the output at BRCk = 0 is “0”. Note that this function can also be defined as shown in (i) of the tenth embodiment.
(C-ii) The individual block gain calculation unit 1180 defines a correction gain value GCk_i for the pixel i (X, Y) based on the color contrast amount BRCk of the block k as in (Equation 25 ). That is, the correction gain value GCk_i has a maximum value MaxRCk at vCenk (CenXk, CenYk), and is defined by a convex function that changes according to the square distance length between vCenk and i. The function for obtaining the correction gain value GCk_i indicates that the possibility of the near view due to the color contrast amount BRCk decreases as it goes to the distant view with vCenk as the center. In addition, in (Formula 25 ), ThDelta indicates the spread of this function, and is constant here regardless of the block, but may be changed according to the size of the block or BRCk. Further, the block color contrast amount and the block brightness contrast amount may be changed.

Next, the following processing is performed on the block brightness contrast amount BRYk of block k.
(Y-i) MaxRYk obtained from the brightness contrast amount BRYk of the block k is set as a correction gain for the center of gravity vCenk (CenXk, CenYk) of the block k. Here, MaxRYk is calculated using a conversion function MaxRYk = GFunc (BRYk) that outputs a value in the range of “0.0” to “1.0”, similarly to the setting of MaxRCk. This function MaxRYk = GFunc (BRYk) is defined by monotonically linearly increasing function output at the maximum value MaxR Y of possible Bryk is "1", the output at Bryk = 0 is "0". This conversion function may be the same as or different from FFunc (BRCk) in (Ci).
(Y-ii) The correction gain value GYk_i for the pixel i (X, Y) based on the brightness contrast amount BRYk of the block k is defined as (Equation 26 ). That is, the correction gain value GYk_i is defined by a convex function having a maximum value MaxRYk at vCenk (CenXk, CenYk) and changing according to the square distance length between vCenk and i. The function for obtaining the correction gain value GYk_i indicates that the possibility of the near view due to the brightness contrast amount BRYk is reduced by moving toward the distant view with vCenk as the center. It should be noted that ThDelta in (Equation 26 ) indicates the spread of this function, and here it is constant regardless of the block, but may be changed according to the size of the block or BRYk. Further, the block color contrast amount and the block brightness contrast amount may be changed.

Finally, the following processing is performed.
(Ti) Using the sub-correction gain GCk_i for the pixel i by the color contrast amount BRCk and the sub-correction gain GYk_i for the pixel i by the brightness contrast amount BRYk in the block k, as shown in (Equation 27 ) The sub correction gain Gk_i for the pixel i based on the two feature quantities in the block k is obtained.

(T-ii) The sum of the sub correction gains Gk_i for the pixel i by all the blocks is obtained, and the obtained value is normalized from the range of “0” to “1” as the correction gain Gi.
(Ci), (C-ii), (Y-i), and (Y-ii) are processed by the individual block gain calculation unit 1180, and the advanced combined gain calculation unit 1181 is (Ti). Process (T-ii) is performed.
FIG. 61 is a diagram for schematically explaining the above processing. As shown in FIG. 61 , in the image processing apparatus of the present embodiment, the possibility of the near view of the pixel i in the case of the color contrast amount BRCk by the block k and the possibility of the close view of the pixel i in the case of the brightness contrast amount BRYk. Then, the sub correction gain Gk_i based on the feature in the block is obtained, and the correction gain Gi of the pixel i is finally obtained in consideration of the influence of all the blocks. Thereby, in the image processing apparatus according to the present embodiment, it is possible to further reduce the influence of fluctuations in the color contrast amount RCi and the brightness contrast amount RYi caused by fluctuations and fluctuations in the color information Ci in the image on the foreground position estimation. it can.

Further, in the image processing apparatus according to the present embodiment, compared with the tenth embodiment, the feature amount in the block is increased to the color contrast and the brightness contrast, thereby improving the estimation accuracy of the correction gain and one feature. It is possible to appropriately deal with patterns that cannot be successfully extracted by quantity.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
[Fifteenth embodiment]
62 to 66 , as a fifteenth embodiment of the present invention, depth information is estimated based on color contrast information, brightness contrast information, and high-frequency component amount in a pixel, and according to the result. An image processing method for correcting color information and an image processing apparatus 1500 will be described.

FIG. 62 shows the configuration of an image processing apparatus 1500 according to the fifteenth embodiment of the present invention.
The image processing apparatus 1500 mainly includes a color information calculation unit 1004, a brightness information calculation unit 1101, a frequency component calculation unit 1201, an extended depth estimation unit 1503, a depth correction unit 1008, and an output unit 1010. Is done.
As shown in FIG. 63 , the extended depth estimation unit 1503 mainly includes a block division unit 1040, an intra-block color average unit 1041, an intra-block brightness average unit 1160, and an intra-block frequency information calculation unit 1300. The block color characteristic calculation unit 1042, the block brightness characteristic calculation unit 1161, the extended foreground block determination unit 1301, the foreground position estimation unit 1031 and the gain calculation unit 1032.
Note that portions similar to those in the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
FIG. 64 shows a process flowchart of an image processing method according to the fifteenth embodiment. FIG. 65 shows a flowchart of depth estimation processing in the image processing method according to the fifteenth embodiment.

As shown in FIG. 62 , the feature of the present embodiment is that in the thirteenth embodiment, the block color contrast amount BRCk related to the average color obtained in the block k and the block contrast amount BRYk related to the average brightness are closer to the foreground block. The three feature values obtained by adding the sum BFk of the high frequency components in the block k to the block color contrast amount BRCk relating to the average color and the block contrast amount BRYk relating to the average brightness obtained in the block k. It is the point which changed into the judgment using.
An image processing method and an image processing apparatus according to the fifteenth embodiment of the present invention will be described.
First, when image data having a pixel value vIi at the pixel i is input to the image processing device 1500, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Conversion processing to brightness information Yi is performed.

  Next, as in the ninth embodiment, the block dividing unit 1040 divides the image to be processed into blocks of a predetermined fixed size, and the intra-block color average unit 1041 is used in each subdivided block k. An average value ACk of color information Ci to be corrected is obtained. The in-block brightness average unit 1160 obtains an average value AYk of brightness information Yi in each subdivided block k. The block color characteristic calculation unit 1042 receives the block inner color average value ACk of the block k from the block inner color average unit 1041, and the block color contrast amount BRCk represented by the ratio of the surrounding block and the average color BCk of the target block k. Ask for. Further, the block brightness characteristic calculation unit 1161 performs the process performed on the pixel unit by the brightness characteristic calculation unit 1110 in the eleventh and twelfth embodiments for each block, and calculates the brightness contrast amount BRYk of the block k. Ask.

  On the other hand, the frequency component calculation unit 1201 calculates the frequency component of the input image vIi. Here, the frequency component calculation unit 1201 performs DCT coefficient calculation to the frequency space by two-dimensional DCT transformation. Usually, the two-dimensional DCT transform is performed on each block after being divided into a predetermined block size due to processing time and the like. Again, two-dimensional DCT conversion to each block is performed in accordance with the block dividing unit 1040 being blocked. Note that the horizontal pixel number npw and the vertical pixel number nph of the block are nph = npw = nnn and are often multiples of 8. Therefore, the block dividing unit 1040 also performs division in this way. If npw of the divided block is not the same as nph, the resolution is converted once into a block image having the same number of pixels, and then two-dimensional DCT conversion is performed. In addition, since it is desirable that each block size is also fixed, this is also done in this embodiment. If the block size is variable, the resolution of each block image is converted to a predetermined block size nf × nf, and then two-dimensional DCT conversion is performed. The DCT coefficients for the pixels in the block k are defined as Fk (s, t) (0 ≦ s, t ≦ nnn−1). Fk (0, 0) indicates a direct current component and indicates a rough feature of the image. Fk (s, t) (s ≠ 0, t ≠ 0) indicates an alternating current component. Here, as s and t are larger, the coefficient indicates a high-frequency component, which corresponds to a minute change (for example, a change in a facial eyelid portion).

The frequency component calculation unit 1201 performs this calculation, and the intra-block frequency information calculation unit 1300 obtains the sum of the coefficients of the frequency components Fk (s, t) in the block k that satisfy s ≧ ThFreq and t ≧ ThFreq. . Fk (s, t) where both s and t are “0” corresponds to information such as the global outline of the image, and corresponds to fine outline information as it goes to the high frequency range. From the sum or average of frequency components equal to or higher than ThFreq, the degree of blur of the in-block image can be known.
Therefore, in the present invention, as schematically shown in FIG. 66 , as a third feature amount, in determining whether the image in the target block k is a blurred region, a predetermined level or higher in the two-dimensional DCT coefficient is used. The average value BFk of DCT coefficients was used (see Equation 28 ).

  Based on the block color contrast amount BRCk, the block brightness contrast amount BRYk, and the frequency component amount BFk determined as the degree of blur, the extended foreground block determination unit 1301 obtains a foreground block candidate group GBr. The determination process here is a process in which the determination is performed by adding the condition that the frequency component amount BFk is larger than the threshold value ThFrq to the condition performed by the integrated foreground block determination unit 1162 in the seventh embodiment. The foreground position estimation unit 1031 obtains the foreground position vCen (CenX, CenY) from the average value of the barycentric coordinates (XBk, YBk) of the selected foreground candidate block k. Thereafter, as in the ninth embodiment, based on the square distance len between the pixel i (X, Y) in the image and the foreground position vCen (CenX, CenY), using (Formula 6), The gain calculation unit 1032 performs processing for obtaining the correction gain value Gi of the pixel i.

In response to this correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
As described above, the image processing apparatus 1500 according to the fifteenth embodiment is characterized in that, as a condition for increasing the determination accuracy, the degree of blur defined by the frequency component amount is added to the foreground block determination condition according to the thirteenth embodiment. is there. As a result, the image processing apparatus 1500 can be expected to improve the accuracy of foreground block candidate selection.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
In the image processing apparatus 1500, the average of DCT coefficients of a certain amount of high-frequency components is used as frequency information in the block. However, the sum can be used, or the sum of DCT coefficients including DC components and AC components can be used. Or an average value thereof may be used.

Further, in the image processing apparatus 1500, two-dimensional DCT transformation is used for frequency component calculation, but frequency transformation such as two-dimensional Fourier transformation (DFT) or two-dimensional wavelet transformation can also be used. Further, the frequency component may be calculated using other orthogonal transforms. Further, the horizontal pixel number npw and the vertical pixel number nph of the block may not be the same.
[Sixteenth Embodiment]
67 and 68 , as a sixteenth embodiment of the present invention, depth information is estimated based on color contrast information, brightness contrast information, and frequency information in a pixel, and color information is determined according to the result. An image processing method and an image processing unit for performing the correction will be described.
FIG. 67 shows the configuration of the extended depth estimation unit 1603 in the image processing apparatus according to the sixteenth embodiment of the present invention. FIG. 68 shows a flowchart of the extended depth estimation process of the image processing method according to the sixteenth embodiment.

The image processing device according to the sixteenth embodiment is obtained by replacing the extended depth estimation unit 1503 with an extended depth estimation unit 1603 in the image processing device 1500 according to the fifteenth embodiment. This is the only difference between the image processing apparatus according to the sixteenth embodiment and the image processing apparatus 1500 according to the fifteenth embodiment, and the other parts are the same. Description is omitted.
The extended depth estimation unit 1603 in the image processing apparatus according to the sixteenth embodiment mainly includes a block division unit 1040, an intra-block color average unit 1041, an intra-block brightness average unit 1160, and a block color characteristic calculation unit 1042. The block brightness characteristic calculation unit 1161, the intra-block frequency information calculation unit 1300, the extended intra-block gain calculation unit 1400, and the extended combined gain calculation unit 1401.

The feature of the present invention is that after obtaining the sub correction gain Gk_i for the pixel i in each block k using the block color contrast amount BRCi and the block brightness contrast amount BRYi for the average color obtained in the block k, In the processing of the fourteenth embodiment, the sum of Gk_i for the pixel i of all the blocks is obtained, and finally the correction gain Gi is used. In addition, frequency information BFk representing the degree of blur is added as a feature amount in each block k. The point is that processing is performed.
An image processing method and an image processing apparatus according to the sixteenth embodiment of the present invention will be described.
First, when image data having the pixel value vIi at the pixel i is input to the image processing apparatus of the present embodiment, the color information calculation unit 1004 converts the data of each pixel constituting the image data vIi into predetermined color information vCi. Conversion processing and conversion processing to brightness information Yi are performed.
Similarly to the fifteenth embodiment, predetermined block division is performed, and the DCT coefficient calculation by two-dimensional DCT transformation is performed by the frequency component calculation unit 1201. Thereafter, the image is divided into blocks of a predetermined fixed size, and an average value ACk of color information Ci that is a correction target in each subdivided block k is obtained by an in-block color averaging unit 1041, and the in-block brightness is obtained. The average unit 1160 obtains an average value AYk of the brightness information Yi in each subdivided block k. Then, the frequency information BFk is obtained by the intra-block frequency information calculation unit 1300. Note that the block division in the block division unit 1040 may be performed according to the block size used in the frequency component calculation unit 1201 or may be performed in another size.

The block color characteristic calculation unit 1042 receives the block inner color average value ACk of the block k from the block inner color average unit 1041, and the block color contrast amount BRCk represented by the ratio of the surrounding block and the average color BCk of the target block k. Ask for.
Also, the block brightness characteristic calculation unit 1161 performs the process performed in units of pixels by the brightness characteristic calculation unit 1110 in the eleventh and twelfth embodiments in units of blocks, and obtains the brightness contrast amount BRYk of the block k.
The expanded intra-block gain calculation unit 1400 uses the sub-correction gain GFk_i for the pixel i based on the frequency information amount BFk as the sub-correction gain value Gk_i for the pixel i based on the feature in the block k shown in FIG. 61 in the fourteenth embodiment . It is determined using Equation 29 ).
The expanded composite gain calculation unit 1401 obtains the product of the sub correction gains GCk_i, GYk_i, and GFk_i using the three feature amounts using (Equation 30 ), and sums the sub correction gain values Gk_i for the pixel i by all the blocks k. And a correction gain value Gi for the final pixel i is obtained.

  Here, MaxFk is a value obtained from the frequency information amount BFk indicating the degree of blur in the block k, and assumes a value from “0.0” to “1.0”.

In response to this correction gain Gi, the depth correction unit 1008 performs predetermined color correction on the target color information Ci, and an image with improved depth feeling is generated.
Finally, the output unit 1010 performs the same processing as in the previous embodiment.
In the image processing apparatus according to the present embodiment, adding the intra-block frequency information representing the degree of blur to the feature amount according to the fourteenth embodiment improves the estimation accuracy of the correction gain, and the feature amount such as color and brightness works well. Patterns that cannot be extracted can also be handled.
In the above description, the processing of the present invention when the target color information Ci is one has been described. For example, in the present invention, color information correction is performed by combining two color information corrections of saturation and hue. It is also possible to execute. In this case, color characteristic information data (color contrast amounts RCi1, RCi2) from the respective color information (Ci1, Ci2) are obtained separately, and correction gains Gi1 and RCi2 corresponding to the color contrast amount RCi1 are obtained. A conversion function for converting into one correction gain Gi based on Gi2 is set.

Further, the block dividing unit 1040 can change the size of each block. In the image processing apparatus according to the present embodiment, the surrounding area in the pixel unit performed in the seventh and eighth embodiments is expanded to correspond to the surrounding block when obtaining the block color contrast amount. May be. In the seventh embodiment and the eighth embodiment, in the present embodiment, the relationship between the pixel and the peripheral region pixel size (the relationship of the number of pixels) (pixel: peripheral region pixel size = 1: N) is matched. The relationship between the size of the block and the area of the surrounding block (block: peripheral area block size = 1: N) may be set.
In the image processing apparatus according to the present embodiment, the average of DCT coefficients of a certain amount of high frequency components is used as frequency information in the block. However, the sum thereof can be used, and a DCT including a DC component and an AC component can be used. You may use the sum total of a coefficient and its average value.

Furthermore, in the image processing apparatus of the present embodiment, two-dimensional DCT transformation is used for frequency component calculation, but frequency transformation such as two-dimensional Fourier transformation (DFT) or two-dimensional wavelet transformation can also be used. Further, the frequency component may be calculated using other orthogonal transforms. Further, the horizontal pixel number npw and the vertical pixel number nph of the block may not be the same.
[Seventeenth embodiment]
An image processing apparatus 1700 according to the seventeenth embodiment shown in FIG. 70 will be described.
The present invention relates to a user mode in which a user selects a processing mode in an apparatus that estimates depth information (near view area / distant view area) from features in an image or corrects color information in an image according to input depth information. A selection unit 3001 is added.
As shown in FIG. 70 , the image processing apparatus 1700 mainly includes an image processing unit 3002 that implements processing in the first to sixteenth embodiments of the present invention, and a display device 3000 that displays the processing results. The user mode selection unit 3001 allows the user to select a processing mode.

  In the present invention, a color correction processing image based on depth estimation performed by the image processing unit 3002 or a color correction processing image based on input depth information is displayed on the display device 3000. This image processing apparatus allows a user to visually check a processed image on the display device 3000 and select a processing mode by a user mode selection unit 3001, and more appropriately according to individual characteristics. The color correction can be performed. Here, the display device 3000 may be a TV main screen or a sub-screen. For example, in the case of a correction effect, the user mode selection unit 3001 preferably uses a menu selection method such as (1) strong mode, (2) medium mode, (3) weak mode, and (4) default mode. In the image processing apparatus according to the present embodiment, when the depth information shown in the first to sixth embodiments is input, the user may specify a foreground area candidate with a pointer such as a mouse. . In the image processing apparatus of the present embodiment, when performing depth estimation in the seventh to sixteenth embodiments, a region specified by the user is set as a foreground region candidate, and the depth estimation unit estimates the foreground position within the range. You may make it use the system to do.

The user confirms the display device 3000, and from the (1) strong mode, (2) medium mode, (3) weak mode, and (4) default mode in the menu for the correction effect prepared in the user mode selection unit 3001. select. In addition, the user designates the foreground candidate area in advance with the mouse.
In response to this selection, the user mode selection unit 3001 performs control of parameters for color correction processing with depth effect applied by the image processing unit 3002, and control for moving the estimated foreground position to a more preferable position. It becomes possible. As a result, in the image processing apparatus of the present embodiment, there is a high possibility that attention will be paid to human visual characteristics, and depth information (foreground position) corresponding to the appearance of each individual can be estimated, and the estimation result It is possible to more appropriately control the correction gain for more obtained color correction.

  Here, an example in which the menu selection method is used for the correction effect in the image processing apparatus of the present embodiment has been described, but the present invention is not limited to this, and switches and buttons corresponding to the correction intensity levels are provided. It may be used. It is also possible to use intensity selection with a continuously moving interface such as a slider bar, which may be an analog moving bar or a digital moving bar. Further, in the image processing apparatus of the present embodiment, even when depth estimation in the seventh to sixteenth embodiments is executed, a method is used in which areas are previously blocked and the corresponding numbers are input / selected. It is also possible. Furthermore, in the image processing apparatus according to the present embodiment, when performing the processes shown in the seventh to ninth embodiments, the first to thirteenth embodiments, and the fifteenth embodiment at the time of depth estimation, a threshold for determining whether or not the subject is a foreground By changing, the estimation result also changes. Therefore, in the image processing apparatus of the present embodiment, for example, the threshold level may be relaxed according to (A) strict mode, (B) intermediate mode, and (C) relaxation mode. In the image processing apparatus of the present embodiment, it is also possible to change the parameter of the correction gain function set based on the obtained near-field position according to the menu.

On the other hand, when the processing of the tenth, fourteenth, and sixteenth embodiments is performed in the image processing apparatus of the present embodiment, a correction gain function for each block is set based on the obtained feature amount, and each correction gain in the target pixel is set. The final correction gain is determined by the sum or product of the functions. In this case, the user can control the depth estimation result by linking the parameters of the correction gain function set in each block with the menu.
[Other Embodiments]
In the first to sixth embodiments, the example in which the depth information fbi is input separately from the image data has been described. However, the depth information fbi is input in addition to the tag information and file format information in the image data. It may be a thing. The depth information input method may also be input from an external memory such as a ROM or a RAM, or may be held in a table prepared in advance. Moreover, the user may input by pointing out with a mouse or a pointer (a foreground point / a distant view point), or the user may input a foreground block number from a block number when the block is roughly divided.

The image processing method and the image processing apparatus of the present invention described in the above embodiment include a processing method and a processing apparatus using color contrast characteristics. Therefore, the effect is greatly influenced by this characteristic.
FIG. 69 shows the characteristics of the color contrast phenomenon, and it has been found that the effect varies depending on the size of the same light yellow circle as the background at the center of the red circle in the light yellow background. For example, when the radius of the light yellow circle at the center is smaller than that of the red circle, the color contrast phenomenon that the light yellow circle at the center of the red circle feels blue is intensified. On the other hand, as the radius of the thin yellow circle at the center increases and approaches the red circle, the color contrast phenomenon that the pale yellow circle at the center of the red circle feels blue is weakened. In addition, in the case of saturation contrast, it is known that the effect varies depending on the size of the same light aqua circle as the background in the blue center portion with high saturation in the light aqua background. In this case, when the radius of the light blue circle with light saturation at the center is small compared to the blue circle with high saturation, the color contrast phenomenon that the saturation of the light blue circle with high saturation at the blue center feels lighter becomes stronger. . On the other hand, as the radius of the thin water circle at the center increases and approaches the blue circle with high saturation, the color contrast phenomenon that the saturation of the thin water circle at the center of the blue circle with high saturation feels lighter becomes weaker. As described above, when the simple patterns having different sizes as shown in FIG. 69 are presented, the image processing method and the image processing apparatus described in the present invention have the characteristic that the color correction effect varies.

  Further, as the color contrast amount data RCi of the target color information, (1) the ratio of the color information of the target pixel to the peripheral representative color information shown in the first to sixteenth embodiments of the present invention, and (2) In addition to the difference between the color information of the target pixel and the peripheral representative color information, it is also possible to use values obtained by converting (1) and (2) with a predetermined function. Similarly, as the brightness contrast amount data RYi of the brightness information, (1) the ratio of the brightness information of the target pixel to the peripheral representative brightness information shown in the fifth to seventh embodiments of the present invention, ( 2) In addition to the difference between the brightness information of the target pixel and the peripheral representative brightness information, it is also possible to use values obtained by converting (1) and (2) with a predetermined function. The predetermined area for the image is a peripheral area for obtaining representative color information and representative brightness information. However, the peripheral area does not need to be fixed. For example, the color Ci of the target pixel i and the pixel k in the peripheral area The peripheral area may be changed adaptively by changing the peripheral area in accordance with the difference between the color information Ck and the distance length between the pixel i and the pixel k. Further, when using the average in the peripheral area as the representative color information, the peripheral color depends on the difference between the color Ci of the target pixel i and the color information Ck of the pixel k in the peripheral area, or the distance length of the pixel i and the pixel k. You may make it weaken or strengthen the weighting coefficient concerning Ck of the area | region pixel k. This setting method for the peripheral region can be similarly applied to the calculation of the brightness contrast amount.

Further, it is assumed that the correction gain Gi of the pixel i estimated by the depth estimation unit in the seventh to sixteenth embodiments takes a value in the range of “0.0” to “1.0”. However, the preset MinGi may be converted to MaxGi (0.0 ≦ MinGi ≦ 1.0, 0.0 ≦ MaxGi ≦ 1.0, MinGi <MaxGi). Further, the MinGi and MaxGi can be designated and input by the user.
In the seventh to sixteenth embodiments of the present invention, the depth correction unit 1008 uses the correction gain Gi of the image i obtained by the depth estimation unit 1006, the second depth estimation unit 1103, and the extended depth estimation unit 1203. The correction target color information Ci in the color information vCi of the input image is corrected. However, it is also possible to correct a plurality of color information in the vCi. In this case, since the color contrast amount RCi is naturally plural, it is processed as vector data such as vRCi.

Further, the correction target color information Ci in the color information vCi of the input image is corrected. However, in addition to the color information, brightness information Yi such as luminance may be corrected using the correction gain Gi.
It is also possible to correct both the color information vCi and the brightness information Yi of the input image at the same time. Further, as in the conventional example (prior art), the intensity in the edge enhancement processing in the image can be controlled by the correction gain Gi, and the shadow addition amount can be controlled in accordance with the correction gain Gi. Is possible. As described above, the correction gain Gi obtained by the depth estimation unit 1006 may be applied to processing for controlling and correcting one or more pieces of monocular information used for human three-dimensional perception.
The parameters used for estimation in each processing device are configured in the processing device, but it is also possible to provide these data with an external memory such as a RAM or an external input unit. is there.

The image processing method and the image processing apparatus of the present invention described in the above embodiments are apparatuses that are built in or connected to devices that handle images, such as computers, televisions, digital cameras, mobile phones, PDAs, and car TVs. It is also realized as an integrated circuit such as an LSI.
Part or all of the functional blocks that implement the various processes described in the above embodiments may be individually made into one chip, or may be made into one chip so as to include some or all of them. Here, although LSI is used, it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
Further, the method of the integrated circuit is not limited to LSI, and can be realized by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI.

Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.
In addition, part or all of the processing of each functional block in each of the above embodiments may be realized by a program. A part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in the computer. In addition, a program for performing each processing is stored in a storage device such as a hard disk or a ROM, and is read out and executed in the ROM or the RAM.
Moreover, each process of the said embodiment may be implement | achieved by hardware, and may be implement | achieved by software. Further, it may be realized by mixed processing of software and hardware.

The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the scope of the invention.
[Appendix]
The present invention can also be expressed as follows.
(Appendix 23)
In an image processing method for correcting input image data,
Estimating depth information of pixels included in the input image, and correcting color information of target pixels based on the estimated depth information;
Image processing method.
(Appendix 24)
The depth information estimation described in the supplementary note 23 is estimated by a color contrast amount obtained by comparing the color information of the target pixel with the color information of the surrounding pixels.
Image processing method.

(Appendix 25)
The depth information estimation described in the supplementary note 23 is performed by comparing the color information of the target pixel with the color information of the surrounding pixels,
Estimated by the brightness contrast amount obtained by comparing the brightness information of the target pixel with the brightness information of the surrounding pixels,
Image processing method.
(Appendix 26)
The depth information estimation described in the supplementary note 23 is performed by comparing the color information of the target pixel with the color information of the surrounding pixels,
The brightness contrast amount obtained by comparing the brightness information of the target pixel with the brightness information of the surrounding pixels,
Estimated by the frequency component of the target pixel value,
Image processing method.

(Appendix 27)
The depth information of the target pixel is defined as a value that is large enough to represent a foreground, and the value defined by a plurality of combinations selected from the color contrast amount, brightness contrast amount, and frequency component of the target pixel is large. The depth information is estimated to show a large value,
The image processing method according to any one of appendices 24, 25, and 26.
(Appendix 28)
The depth information of the target pixel is defined as a value that is small enough to represent a distant view, and the value defined by a plurality of combinations selected from the color contrast amount, brightness contrast amount, and frequency component of the target pixel is small. The depth information is estimated to show a small value,
The image processing method according to any one of appendices 24, 25, and 26.

(Appendix 29)
In a color image processing method for correcting input color image data,
A color information calculation step for calculating color information of the input image signal;
A depth estimation step of estimating depth information from the color information;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing method comprising:
(Appendix 30)
The depth estimation step obtains a color contrast amount obtained by comparing the color information of the target pixel with the surrounding color information representing the color information in the surrounding area,
Depth degree is estimated based on the color contrast amount,
According to the estimated depth degree, a correction gain value for each pixel is calculated.
The image processing method according to attachment 29.

(Appendix 31)
The depth estimation step obtains a color contrast amount obtained by comparing the color information of the target pixel with the surrounding color information representing the color information in the surrounding area,
Based on the color contrast amount, determine the pixels included in the foreground,
The foreground position is estimated from the judgment result,
According to the estimated foreground position, a correction gain value for each pixel is calculated.
The image processing method according to attachment 29.
(Appendix 32)
The depth estimation step performs block object division in the image,
Find the average color information in each block,
The block color contrast amount obtained by comparing the average color information of the target block with the surrounding block average color information representing the average color information in a plurality of blocks around the block,
Based on the block color contrast amount, the block included in the foreground is determined,
The foreground position is estimated from the judgment result,
According to the estimated foreground position, a correction gain value for each pixel is calculated.
The image processing method according to attachment 29.

(Appendix 33)
The depth estimation step performs block object division in the image,
Find the average color information in each block,
The block color contrast amount obtained by comparing the average color information of the target block with the surrounding block average color information representing the average color information in a plurality of blocks around the block,
Calculate the correction gain value for each block based on the block color contrast amount,
The correction gain value of each pixel is calculated by combining the correction gain values from all blocks for the target pixel to obtain the correction gain value.
The image processing method according to attachment 29.
(Appendix 34)
In a color image processing method for correcting input color image data,
A color information calculation step for calculating color information of the input image signal;
A brightness information calculating step for calculating brightness information of the input image signal;
An altitude depth estimation step of estimating depth information from the color information and brightness information;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing method comprising:

(Appendix 35)
The altitude depth estimation step
A color characteristic calculating step for obtaining a color contrast amount of a target pixel;
A brightness characteristic calculating step for obtaining a brightness contrast amount of the target pixel;
An altitude depth degree calculating step for estimating a depth degree from the color contrast amount and the brightness contrast amount is performed.
The image processing method according to attachment 34.
(Appendix 36)
The altitude depth estimation step
A color characteristic calculating step for obtaining a color contrast amount of a target pixel;
A brightness characteristic calculating step for obtaining a brightness contrast amount of the target pixel;
Judge the pixels included in the foreground from the color contrast amount and brightness contrast amount,
The foreground position is estimated from the judgment result,
According to the estimated foreground position, a correction gain value for each pixel is calculated.
The image processing method according to attachment 34.

(Appendix 37)
The altitude depth estimation step performs block object division in the image,
Find the average color information and average brightness information in each block,
Find the block color contrast amount for the average color information of the target block,
Find the block brightness contrast amount for the average brightness information of the target block,
Based on the block color contrast amount and block brightness contrast amount, the block included in the foreground is determined,
The foreground position is estimated from the judgment result,
According to the estimated foreground position, a correction gain value for each pixel is calculated.
The image processing method according to attachment 34.

(Appendix 38)
The altitude depth estimation step performs block object division in the image,
Find the average color information and average brightness information in each block,
Find the block color contrast amount for the average color information of the target block,
Find the block brightness contrast amount for the average brightness information of the target block,
Based on the block color contrast amount and block brightness contrast amount, calculate the correction gain value for each block,
The correction gain value of each pixel is calculated by combining the correction gain values from all blocks for the target pixel to obtain the correction gain value.
The image processing method according to attachment 34.

(Appendix 39)
In a color image processing method for correcting input color image data,
A color information calculation step for calculating color information of the input image signal;
A brightness information calculating step for calculating brightness information of the input image signal;
A frequency component calculating step for calculating a frequency component of the input image signal;
Extended depth estimation step for estimating depth information from the color information, brightness information, and frequency components;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing method comprising:

(Appendix 40)
The extended depth estimation step performs block object division in the image,
Find the average color information and average brightness information in each block,
Find the frequency component information in each block,
Find the block color contrast amount for the average color information of the target block,
Find the block brightness contrast amount for the average brightness information of the target block,
Based on the block color contrast amount, block brightness contrast amount, and intra-block frequency components, the block included in the foreground is determined.
The foreground position is estimated from the judgment result,
According to the estimated foreground position, a correction gain value for each pixel is calculated.
40. The image processing method according to appendix 39.

(Appendix 41)
The extended depth estimation step performs block object division in the image,
Find the average color information and average brightness information in each block,
Find the frequency component information in each block,
Find the block color contrast amount for the average color information of the target block,
Find the block brightness contrast amount for the average brightness information of the target block,
Calculate the correction gain value by each block based on the block color contrast amount, block brightness contrast amount, and intra-block frequency component,
The correction gain value of each pixel is calculated by combining the correction gain values from all blocks for the target pixel to obtain the correction gain value.
40. The image processing method according to appendix 39.

(Appendix 42)
In a color image processing method for correcting input color image data,
A user process selection step in which the user selects a process mode from a menu or button presented by a predetermined method;
Performing process control of the image processing method according to any one of appendices 29, 34, and 39 according to the processing mode obtained in the user process selection step;
Displaying the result of the image processing method on a predetermined display device;
If the processed image obtained in the display step is good, a user output instruction step for instructing to output the image by a predetermined method;
An output step of outputting a processed image by a predetermined method when an output instruction is given in the user output step;
Further comprising
40. The image processing method according to any one of appendices 29, 34, and 39.

(Appendix 43)
In the user processing selection step, the intensity at the time of performing color correction in the processing in the image processing method according to any one of Supplementary Notes 29, 34, and 39 is selected from a plurality of modes set as strong mode, medium mode, and weak mode. select,
45. The image processing method according to appendix 42.
(Appendix 44)
The color correction in the image processing method according to any one of appendices 29, 34, and 39 operates so that the effect increases or decreases according to a change in the size of a predetermined pattern in which a color contrast phenomenon occurs.
40. The image processing method according to any one of appendices 29, 34, and 39.
(Appendix 45)
In a color image processing apparatus that corrects input color image data,
A color information calculation unit for calculating color information of the input image signal;
A depth estimation unit for estimating depth information from the color information;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An output unit that outputs the corrected color information obtained by the depth correction unit as predetermined image data;
An image processing apparatus comprising:

(Appendix 46)
In a color image processing apparatus that corrects input color image data,
A color information calculation unit for calculating color information of the input image signal;
A brightness information calculation unit for calculating brightness information of the input image signal;
An altitude depth estimation unit that estimates depth information from the color information and brightness information;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An output unit that outputs the corrected color information obtained by the depth correction unit as predetermined image data;
An image processing apparatus comprising:
(Appendix 47)
In a color image processing apparatus that corrects input color image data,
A color information calculation unit for calculating color information of the input image signal;
A brightness information calculation unit for calculating brightness information of the input image signal;
A frequency component calculator that calculates the frequency component of the input image signal;
An extended depth estimation unit that estimates depth information from the color information, brightness information, and frequency components;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An output unit that outputs the corrected color information obtained by the depth correction unit as predetermined image data;
An image processing apparatus comprising:

(Appendix 48)
In a color image processing apparatus that corrects input color image data,
A user process selection unit for the user to select a process mode from a menu or button presented by a predetermined method;
The image processing unit according to any one of appendices 23 to 41, which performs predetermined image processing according to a processing mode obtained by the user processing selection unit;
A display unit for displaying a result of the image processing unit on a predetermined display device;
An image processing apparatus comprising:
(Appendix 49)
An image processing program for correcting a color image by a computer,
The image processing program includes:
A color information calculation step for calculating color information of the input image signal;
A depth estimation step of estimating depth information from the color information;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing program that is an image processing method.

(Appendix 50)
An image processing program for correcting a color image by a computer,
The image processing program includes:
A color information calculation step for calculating color information of the input image signal;
A brightness information calculating step for calculating brightness information of the input image signal;
An altitude depth estimation step of estimating depth information from the color information and brightness information;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing program that is an image processing method.

(Appendix 51)
An image processing program for correcting a color image by a computer,
The image processing program includes:
A color information calculation step for calculating color information of the input image signal;
A brightness information calculating step for calculating brightness information of the input image signal;
A frequency component calculating step for calculating a frequency component of the input image signal;
Extended depth estimation step for estimating depth information from the color information, brightness information, and frequency components;
A depth correction step for performing depth correction on color information in the image based on the depth information;
An output step of outputting the corrected color information obtained in the depth correction step as predetermined image data;
An image processing program that is an image processing method.

(Appendix 52)
A color information calculation unit for calculating color information of the input image signal;
A depth estimation unit for estimating depth information from the color information;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An integrated circuit comprising: an output unit that outputs corrected color information obtained by the depth correction unit as predetermined image data.
(Appendix 53)
A color information calculation unit for calculating color information of the input image signal;
A brightness information calculation unit for calculating brightness information of the input image signal;
A second depth estimation unit that estimates depth information from the color information and brightness information;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An output unit that outputs the corrected color information obtained by the depth correction unit as predetermined image data;
An integrated circuit comprising:

(Appendix 54)
A color information calculation unit for calculating color information of the input image signal;
A brightness information calculation unit for calculating brightness information of the input image signal;
A frequency component calculator that calculates the frequency component of the input image signal;
An extended depth estimation unit that estimates depth information from the color information, brightness information, and frequency components;
A depth correction unit that performs depth correction on color information in the image based on the depth information;
An output unit that outputs the corrected color information obtained by the depth correction unit as predetermined image data;
An integrated circuit comprising:

  The image processing apparatus according to the present invention changes the color information (saturation, hue, etc.) according to the depth information indicating the near view / distant view, thereby improving the depth perception of the 2D image in a simple and 3D image display manner. Since it can be realized without requiring a special device, it is useful in the field of video equipment-related industries, and the image processing apparatus and the like according to the present invention can be implemented in this field.

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention. The processing flowchart figure of the image processing method which concerns on 1st Embodiment of this invention. The figure which shows the example of depth information typically. The figure which shows the example of a control coefficient which performs the saturation correction | amendment according to the depth information in this invention. 1 is a block diagram showing a configuration of a color information correction unit in an image processing apparatus according to a first embodiment of the present invention. The flowchart figure of the process of the color information correction | amendment part of the image processing method which concerns on 1st Embodiment of this invention. The figure which shows the outline | summary of the color contrast which is the characteristics of invention of 1st Embodiment in this invention. The figure which shows typically the outline | summary of the saturation correction | amendment using a saturation contrast. The figure which shows typically the outline | summary of the hue correction | amendment using a color relative ratio. The figure which represents typically control of the saturation correction | amendment using the saturation contrast by the correction control parameter by depth information in the image processing method which concerns on 1st Embodiment in this invention. The block diagram which shows the structure of the color information correction | amendment part in the image processing apparatus which concerns on 2nd Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 2nd Embodiment in this invention. The figure which shows the fluctuation | variation of the correction coefficient which controls the contrast effect in the saturation correction by the invention which concerns on 2nd Embodiment. The block diagram which shows the structure of the image processing apparatus which concerns on 3rd Embodiment in this invention. The block diagram which shows the structure of the hue corresponding | compatible color information correction | amendment part in the image processing apparatus which concerns on 3rd Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 3rd Embodiment in this invention. The figure which shows the fluctuation | variation of the correction coefficient which controls the contrast effect within the saturation correction by the invention which concerns on 3rd Embodiment (skin color area | region object). The figure which shows the fluctuation | variation of the correction coefficient which controls the contrast effect in the saturation correction by the invention which concerns on 3rd Embodiment (sky, green area object). The block diagram which shows the structure of the image processing apparatus which concerns on 4th Embodiment in this invention. The block diagram which shows the structure of the brightness corresponding | compatible color information correction | amendment part in the image processing apparatus which concerns on 4th Embodiment in this invention. The processing flowchart figure of the image processing method which is invention which concerns on 4th Embodiment in this invention. The figure which shows the fluctuation | variation of the correction coefficient which controls the contrast effect in the saturation correction by the invention which concerns on 4th Embodiment. The block diagram which shows the structure of the image processing apparatus which concerns on 5th Embodiment in this invention. The block diagram which shows the structure of the brightness characteristic calculation part in the image processing apparatus which concerns on 5th Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 5th Embodiment in this invention. The process flowchart figure of the brightness characteristic calculation step in the image processing method which concerns on 5th Embodiment in this invention. The figure which shows the fluctuation | variation of the correction coefficient which controls the contrast effect in the saturation correction by the invention which concerns on 5th Embodiment. The block diagram which shows the structure of the image processing apparatus which concerns on 6th Embodiment in this invention. The block diagram which shows the structure of the extended color information correction | amendment part in the image processing apparatus which concerns on 6th Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 6th Embodiment in this invention. The block diagram which shows the structure of the image processing apparatus which concerns on 7th Embodiment in this invention. The block diagram which shows the structure of the depth estimation part of the image processing apparatus which concerns on 7th Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 7th Embodiment in this invention. The processing flowchart figure of the depth information estimation step in the image processing method concerning 7th Embodiment in this invention. The figure which represents typically the outline | summary of the depth estimation algorithm in the image processing method which concerns on 7th Embodiment in this invention. The function figure which shows the example of control of the correction | amendment gain value obtained from the estimated depth information. The block diagram which shows the structure of the depth estimation part of the image processing apparatus which concerns on 8th Embodiment in this invention. The processing flowchart figure of the depth information estimation step in the image processing method which concerns on 8th Embodiment in this invention. The figure which shows typically the correction gain implemented at the depth estimation step in the image processing method which concerns on 8th Embodiment in this invention. The block diagram which shows the structure of the depth estimation part of the image processing apparatus which concerns on 9th Embodiment in this invention. The processing flowchart figure of the depth estimation step in the image processing method which concerns on 9th Embodiment in this invention. The figure which shows typically the feature-value (color contrast) used by the depth estimation by the block division by the image processing method which concerns on 9th Embodiment in this invention. The block diagram which shows the structure of the depth estimation part in the image processing apparatus which concerns on 10th Embodiment in this invention. The processing flowchart figure of the depth estimation step of the image processing method which concerns on 10th Embodiment in this invention. The schematic diagram which shows a mode that the correction gain set in each block is integrated and the correction gain in a target pixel is determined. The block diagram which shows the structure of the image processing apparatus which concerns on 11th Embodiment in this invention. The block diagram which shows the structure of the height depth estimation part in the image processing apparatus which concerns on 11th Embodiment in this invention. The processing flowchart figure of the depth information estimation step in the image processing method which concerns on 11th Embodiment in this invention. The processing flowchart figure of the altitude depth estimation step in the image processing method which concerns on 11th Embodiment in this invention. The processing flowchart figure of the color contrast amount calculation step and the brightness contrast amount calculation step in the image processing method according to the eleventh embodiment of the present invention. The figure which shows the outline | summary of the brightness contrast used by this invention. The figure which shows the control example of the correction | amendment gain value obtained from the estimated depth information. The block diagram which shows the structure of the altitude depth estimation part in the image processing apparatus which concerns on 12th Embodiment in this invention. The processing flowchart figure of the altitude depth information estimation step in the image processing method which concerns on 12th Embodiment in this invention. The figure which shows typically a foreground position estimation and the correction gain function obtained from there. The block diagram which shows the structure of the altitude depth estimation part in the image processing apparatus which concerns on 13th Embodiment in this invention. The processing flowchart figure of the altitude depth information estimation step in the image processing method concerning 13th Embodiment in this invention. The figure which shows typically the feature-value (color contrast and brightness contrast) used for depth estimation from each block. The block diagram which shows the structure of the altitude depth information estimation part in the image processing apparatus which concerns on 14th Embodiment in this invention. The processing flowchart figure of the altitude depth estimation step of the image processing method which concerns on 14th Embodiment in this invention. The schematic diagram which shows a mode that the correction gain set in each block is integrated and the correction gain in a target pixel is determined. The block diagram which shows the structure of the image processing apparatus which concerns on 15th Embodiment in this invention. The block diagram which shows the structure of the extended depth estimation part in the image processing apparatus which concerns on 15th Embodiment in this invention. The processing flowchart figure of the image processing method which concerns on 15th Embodiment in this invention. The processing flowchart figure of the extended depth information estimation step in the image processing method which concerns on 15th Embodiment in this invention. The figure which shows typically the feature-value (a color contrast, a brightness contrast, a frequency component) used for depth estimation from each block. The block diagram which shows the structure of the extended depth estimation part in the image processing apparatus which concerns on 16th Embodiment in this invention. The processing flowchart figure of the extended depth information estimation step in the image processing method which concerns on 16th Embodiment in this invention. The figure for showing the fluctuation | variation of the color contrast effect in the image processing method in this invention. The figure which shows the structure of the image processing apparatus which concerns on 17th Embodiment of this invention. FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus according to a conventional example 1. FIG. 6 is a block diagram showing a configuration of an image processing apparatus of Conventional Example 2.

Explanation of symbols

100, 300, 400, 600, 700, 1100, 1500, 1700 Image processing apparatus 3002 Image processing unit 10 Correction control unit 11 Color information calculation unit 12, 2012 Color information correction unit 13 Output unit 20 Color characteristic calculation unit 21 Correction amount calculation Units 22, 2022, 3022, 4022, 5022, 6022 correction unit 23 peripheral color information calculation unit 24 color contrast amount calculation units 30, 3030, 5030, 6030 contrast correction amount control unit 40 hue calculation unit 42 hue corresponding color information correction unit 50 Brightness information calculation unit 52 Brightness corresponding color information correction unit 60 Brightness characteristic calculation unit 70 Ambient brightness information calculation unit 71 Brightness contrast amount calculation unit 80 Extended color information correction unit 1002 Image processing unit 1004 Color information calculation unit 1006 806, 906, 10006, depth estimation unit 1008 depth correction unit 1010 output unit 1020 Calculation unit 1021 Peripheral representative color calculation unit 1022 Color contrast amount calculation unit 1023 Depth degree calculation unit 1030 Foreground pixel determination unit 1031 Foreground position estimation unit 1032 Gain calculation unit 1040 Block division unit 1041 In-block average unit 1042 Block color characteristic calculation unit 1043 Block determination unit 1044 Block peripheral representative color calculation unit 1045 Block color contrast amount calculation unit 1050 In-block gain calculation unit 1051 Composite gain calculation unit 1101 Brightness information calculation units 1103, 1203, 1303, 1403, 1503, 1603 Second depth estimation unit 1110 brightness characteristic calculation unit 1111 around the representative brightness calculation unit 1112 brightness contrast amount calculation unit 1113 integrates the depth calculation unit 1130 integrates foreground pixel determining unit 1160 the block brightness averaging unit 1161 block brightness characteristic calculation unit 116 Integrated foreground block determination unit 1180 Individual block gain calculation unit 1181 Advanced composite gain calculation unit 1200 Third image processing unit 1201 Frequency component calculation unit 1202 Frequency information 1203 Extended depth estimation unit 1300 In-block frequency information calculation unit 1301 Extended foreground block determination unit 1400 Extended block gain calculator 1401 Extended combined gain calculator 3000 Display device (display unit)
3001 User mode selection section

Claims (17)

A color information calculation unit that calculates color information of the image signal from an image signal capable of forming an image composed of pixels;
A correction amount control unit for obtaining a correction gain for correcting the color information of the image signal based on the input depth information;
The color information is corrected based on the color information based on the color information of the pixel of interest that is the pixel to be processed in the image signal, the color contrast amount based on the color information of the peripheral pixel of the pixel of interest, and the correction gain. A color information correction unit to perform,
An output unit that outputs the color information corrected by the color information correction unit in a predetermined format;
Images processing device provided with a.   The color information correction unit increases the color contrast effect between the color information of the pixel of interest and the color information of the surrounding pixels of the pixel of interest as the degree of the depth information regarding the pixel of interest is large. By performing the correction so that the sense of distant view is enhanced by the color contrast effect as the degree of indicating that the sense of distant view is increased and the depth information about the pixel of interest is a distant view.
  The image processing apparatus according to claim 1. The color information correction unit
A color characteristic calculating unit for calculating a pre-Symbol color contrast amount,
A correction amount calculation unit that calculates a contrast correction coefficient for correcting the color information based on the correction gain and the color contrast amount;
A correction unit that corrects the color information based on the contrast correction coefficient;
Having
The image processing apparatus according to claim 2 . The color characteristic calculation unit calculates a ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and a representative value ACi of the color information of the peripheral pixels as the color contrast amount;
The correction amount calculation unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain . When the color contrast amount RCi is “1”, the contrast correction coefficient KLi is set to “0”. And setting the contrast correction coefficient KLi so that the contrast correction coefficient KLi is monotonous with respect to the color contrast amount RCi,
The correction unit obtains the color information correction amount dCi by controlling the contrast correction coefficient KLi, the color information Ci, and the constant α1, and determines the corrected color information Ci_new after correction of the target pixel as Ci_new = (the color Information Ci) + (color information correction amount dCi) to correct the color information of the pixel of interest;
The image processing apparatus according to claim 3 . The color characteristic calculator calculates a ratio RSi (= Si / ASi) between the saturation Si of the target pixel and the representative value ASi of the saturation of the surrounding pixels as the color contrast amount,
The correction amount calculation unit determines a first foreground threshold (> 0) when the degree of the correction gain for the pixel of interest is the maximum and the color contrast amount is greater than “1”. The contrast correction coefficient Ki is set to a smaller positive value, and when the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second foreground threshold (<0). When the correction gain for the pixel of interest is the maximum indicating that it is a distant view, and when the color contrast amount is greater than “1”, the first distant view value is smaller than the first foreground threshold. When the contrast correction coefficient Ki is set to a positive value smaller than a threshold value (> 0) and the color contrast amount is “1” or less, a second far-view threshold value (< 0) The contrast correction coefficient Ki is set to a larger negative value. And,
The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the constant α2, and the corrected color information Si_new after the correction of the pixel of interest is Si_new = (the saturation). Degree Si) + (the saturation correction amount dSi) to correct the color information of the pixel of interest.
The image processing apparatus according to claim 4 . The color information correction unit
A color characteristic calculating unit for calculating a pre-Symbol color contrast amount,
A contrast correction amount control unit for obtaining a contrast correction coefficient control amount α4 based on the color contrast amount and the color information;
A correction amount calculation unit that calculates a contrast correction coefficient for correcting the color information based on the correction gain and the color contrast amount;
A correction unit that corrects the color information based on the contrast correction coefficient and the contrast correction coefficient control amount;
Having
The image processing apparatus according to claim 2 . The color information correction unit
A brightness characteristic calculator that calculates a brightness contrast amount based on the brightness information of the target pixel and the brightness information of the surrounding pixels;
The contrast correction amount control unit obtains the contrast correction coefficient control amount α4 based on the color contrast amount and the brightness contrast amount.
The image processing apparatus according to claim 6 . The color characteristic calculation unit calculates a ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and a representative value ACi of the color information of the peripheral pixels as the color contrast amount;
The correction amount calculation unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain . When the color contrast amount RCi is “1”, the contrast correction coefficient KLi is set to “0”. And setting the contrast correction coefficient KLi so that the contrast correction coefficient KLi is monotonous with respect to the color contrast amount RCi,
The correction unit obtains the color information correction amount dCi by controlling with the contrast correction coefficient KLi, the color information Ci, and the contrast correction coefficient control amount α4, and the corrected color information Ci_new after the correction of the target pixel is obtained. By correcting Ci_new = (color information Ci) + (color information correction amount dCi), the color information of the target pixel is corrected.
The image processing apparatus according to claim 6 . The color characteristic calculation unit calculates a ratio RCi (= Ci / ACi) between the color information Ci of the pixel of interest and a representative value ACi of the color information of the peripheral pixels as the color contrast amount;
The correction amount calculation unit determines an upper limit value and a lower limit value of the contrast correction coefficient KLi based on the correction gain . When the color contrast amount RCi is “1”, the contrast correction coefficient KLi is set to “0”. And setting the contrast correction coefficient KLi so that the contrast correction coefficient KLi is monotonous with respect to the color contrast amount RCi,
The correction unit obtains the color information correction amount dCi by controlling with the contrast correction coefficient KLi, the color information Ci, and the contrast correction coefficient control amount α4, and the corrected color information Ci_new after the correction of the target pixel is obtained. By correcting Ci_new = (color information Ci) + (color information correction amount dCi), the color information of the target pixel is corrected.
The image processing apparatus according to claim 7 . The color characteristic calculator calculates a ratio RSi (= Si / ASi) between the saturation Si of the target pixel and the representative value ASi of the saturation of the surrounding pixels as the color contrast amount,
The correction amount calculation unit determines a first foreground threshold (> 0) when the degree of the correction gain for the pixel of interest is the maximum and the color contrast amount is greater than “1”. The contrast correction coefficient Ki is set to a smaller positive value, and when the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second foreground threshold (<0). When the correction gain for the pixel of interest is the maximum indicating that it is a distant view, and when the color contrast amount is greater than “1”, the first distant view value is smaller than the first foreground threshold. When the contrast correction coefficient Ki is set to a positive value smaller than a threshold value (> 0) and the color contrast amount is “1” or less, a second far-view threshold value (< 0) The contrast correction coefficient Ki is set to a larger negative value. And,
The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the contrast correction coefficient control amount α4, and the corrected color information Si_new after the correction of the target pixel is obtained. Si_new = (saturation Si) + (saturation correction amount dSi) to correct the color information of the target pixel.
The image processing apparatus according to claim 8 . The color characteristic calculator calculates a ratio RSi (= Si / ASi) between the saturation Si of the target pixel and the representative value ASi of the saturation of the surrounding pixels as the color contrast amount,
The correction amount calculation unit determines a first foreground threshold (> 0) when the degree of the correction gain for the pixel of interest is the maximum and the color contrast amount is greater than “1”. The contrast correction coefficient Ki is set to a smaller positive value, and when the color contrast amount is “1” or less, the contrast correction coefficient Ki is set to a negative value larger than the second foreground threshold (<0). When the correction gain for the pixel of interest is the maximum indicating that it is a distant view, and when the color contrast amount is greater than “1”, the first distant view value is smaller than the first foreground threshold. When the contrast correction coefficient Ki is set to a positive value smaller than a threshold value (> 0) and the color contrast amount is “1” or less, a second far-view threshold value (< 0) The contrast correction coefficient Ki is set to a larger negative value. And,
The correction unit obtains the saturation correction amount dSi by controlling the contrast correction coefficient Ki, the saturation Si, and the contrast correction coefficient control amount α4, and the corrected color information Si_new after the correction of the target pixel is obtained. Si_new = (saturation Si) + (saturation correction amount dSi) to correct the color information of the target pixel.
The image processing apparatus according to claim 9 . A color information calculation step of calculating color information of the image signal from an image signal capable of forming an image composed of pixels;
A correction amount control step for obtaining a correction gain for correcting the color information of the image signal based on the input depth information;
The color information is corrected based on the color information based on the color information of the pixel of interest that is the pixel to be processed in the image signal, the color contrast amount based on the color information of the peripheral pixel of the pixel of interest, and the correction gain. Color information correction step to perform,
The image processing method having a. Computer
A color information calculation unit for calculating color information of the image signal from an image signal capable of forming an image composed of pixels;
A correction amount control unit for obtaining a correction gain for correcting the color information of the image signal based on the input depth information;
The color information is corrected based on the color information based on the color information of the pixel of interest that is the pixel to be processed in the image signal, the color contrast amount based on the color information of the peripheral pixel of the pixel of interest, and the correction gain. Color information correction unit
Program to function as. Computer
A color information calculation unit for calculating color information of the image signal from an image signal capable of forming an image composed of pixels;
A correction amount control unit for obtaining a correction gain for correcting the color information of the image signal based on the input depth information;
The color information is corrected based on the color information based on the color information of the pixel of interest that is the pixel to be processed in the image signal, the color contrast amount based on the color information of the peripheral pixel of the pixel of interest, and the correction gain. Color information correction unit
A computer-readable recording medium on which a program for functioning as a computer is recorded. A color information calculation unit that calculates color information of the image signal from an image signal capable of forming an image composed of pixels;
A correction amount control unit for obtaining a correction gain for correcting the color information of the image signal based on the input depth information;
The color information is corrected based on the color information based on the color information of the pixel of interest that is the pixel to be processed in the image signal, the color contrast amount based on the color information of the peripheral pixel of the pixel of interest, and the correction gain. A color information correction unit to perform,
Bei obtain integrated circuit. A user mode selection unit for selecting a processing mode according to a user instruction;
A display unit for displaying an output from the output unit as an image;
Further comprising
The image processing apparatus according to claim 1. The user mode selection unit is configured to select at least the processing mode including information on the intensity of correction of the color information;
The image processing apparatus according to claim 16 .

Images