JP6413210B2 - Image processing apparatus, imaging apparatus, and program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and a program.

  Conventionally, various methods for creating an image to be output to a monitor, a printer, or the like from an image taken with an electronic camera or the like have been considered. For example, when creating an image for a standard sRGB monitor, a sensor RGB such as an electronic camera is directly used as an output color space (for example, AdobeRGB in addition to sRGB) using a 3 × 3 color conversion matrix. If converted, a negative value or an overflow value occurs in a high saturation portion that cannot be reproduced in the output color space. Further, when these values are clipped at 0 value or the maximum display value, a so-called gradation step where the gradation changes suddenly may become conspicuous. Therefore, in the invention of Patent Document 1, values before matrix conversion are used in accordance with the output so that values before matrix conversion are used in the minus region and overflow region, and values after matrix conversion are used in other regions. A technique for reducing a gradation step by performing a process of weighted averaging of values after matrix conversion is disclosed.

JP 2008-219271 A

  However, in the invention of Patent Document 1, as described above, the value in the color space before matrix conversion and the value in the output color space after matrix conversion are weighted and averaged. Here, the value in the color space before the matrix conversion and the value in the output color space after the matrix conversion are largely separated, and the saturation and hue change in the portion where these values are averaged. There was a problem.

  The present invention has been made in view of the above problems, and an object of the present invention is to realize suitable color reproducibility without changing saturation and hue in signal conversion of a color image.

The image processing apparatus as an example of the present invention includes an acquisition unit that acquires a signal of a color image output from the image pickup device, a signal conversion unit for converting the signal of the color image into a luminance signal, noted in the color image A luminance adjustment unit that adjusts the luminance signal of the target pixel according to the saturation and hue of the pixel, and the luminance adjustment unit reduces the luminance so that the luminance decreases as the saturation of the target pixel increases. Find the amount of signal adjustment .

The signal conversion unit converts the color image signal into a luminance signal and a color difference signal, and the luminance adjustment unit divides the color difference plane into at least three color regions according to hues, and a different coefficient for each color region. , The color difference signal of the target pixel of the color image may be mapped onto a color difference plane, and the adjustment amount may be obtained using the coefficient of the color region to which the color difference signal belongs.

  The luminance adjustment unit may obtain the adjustment amount by normalizing the value of the color difference signal with the value of the luminance signal.

  In addition, the luminance adjustment unit divides the color difference plane based on the standardized color difference signal into at least three color regions for each hue, and associates different coefficients with each color region, so that the color difference of the target pixel of the color image The signal may be mapped onto a color difference plane, and the adjustment amount may be obtained using a coefficient of a color region to which the color difference signal belongs.

  An imaging apparatus that is an example of the present invention includes an imaging unit that captures a color image and the above-described image processing apparatus.

Program is an example of the present invention, a process for acquiring the signal of the color image output from the imaging device, and a process of converting a signal of the color image into a luminance signal, chroma and the target pixel of the color image A process of adjusting the luminance signal of the pixel of interest in accordance with the hue , and the process of adjusting the luminance signal in the computer, so that the luminance decreases as the saturation of the pixel of interest increases. A process for obtaining the adjustment amount of the signal is performed.

  According to the present invention, it is possible to realize suitable color reproducibility without changing saturation or hue in signal conversion of a color image.

The figure which shows the structural example of the electronic camera of 1st Embodiment. 1 is a flowchart showing an operation example of the electronic camera of the first embodiment. The figure which shows the gradation conversion curve G1 applied by the 1st gradation conversion process The figure which shows the example of division of the color difference plane The figure which shows the gradation conversion curve G2 applied by a 2nd gradation conversion process Another figure showing an example of color difference plane division The figure which shows the structural example of the image processing apparatus in 3rd Embodiment.

<Description of First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of the electronic camera according to the first embodiment which is an example of an image processing apparatus and an imaging apparatus.

  The electronic camera 11 includes a photographing optical system 12, an imaging unit 13, an image processing engine 14, a first memory 15, a second memory 16, a recording I / F 17, a display unit 18, and an operation unit 19. I have. Here, the imaging unit 13, the first memory 15, the second memory 16, the recording I / F 17, the display unit 18, and the operation unit 19 are each connected to the image processing engine 14.

  The imaging unit 13 is a module that captures (captures) an image of a subject formed by the imaging optical system 12. For example, the imaging unit 13 includes an imaging device that performs photoelectric conversion, and a signal processing circuit that performs analog signal processing, A / D conversion processing, and the like on the output of the imaging device. Here, for example, RGB color filters are arranged in the pixels of the image sensor according to a known Bayer array, and image signals corresponding to the respective colors are output by color separation in the color filters. Thereby, the imaging part 13 can acquire a color image at the time of imaging | photography. Note that image data captured by the imaging unit 13 is input to the image processing engine 14.

  The image processing engine 14 is a processor that comprehensively controls the operation of the electronic camera 11, and performs image processing such as color interpolation, gradation conversion, white balance correction, edge enhancement, and noise removal on image data. Apply. The image processing engine 14 functions as a signal conversion unit 21 and a luminance adjustment unit 22 by executing a program (the operations of the signal conversion unit 21 and the luminance adjustment unit 22 will be described later).

  The first memory 15 is a memory that temporarily stores data of each image, and is configured by, for example, an SDRAM that is a volatile storage medium. The second memory 16 is a memory that stores a program executed by the image processing engine 14 and various data used in the program, and is configured by a non-volatile memory such as a flash memory, for example.

  The recording I / F 17 has a connector for connecting the nonvolatile storage medium 20. The recording I / F 17 executes writing / reading of image data with respect to the storage medium 20 connected to the connector. The storage medium 20 is, for example, a hard disk or a memory card incorporating a semiconductor memory. In FIG. 1, a memory card is illustrated as an example of the storage medium 20.

  The display unit 18 is a display device that displays various images. The display unit 18 performs, for example, live view display of an image based on the output of the imaging unit 13 and reproduction display based on an output from the storage medium 20. In addition, the operation unit 19 includes a switch that receives various user operations (for example, an instruction to shoot a subject, an instruction to change setting of brightness adjustment, and the like).

  Next, an operation example of the electronic camera of the first embodiment will be described with reference to the flowchart of FIG. In the first embodiment, an example of adjusting the luminance of a color image will be described. 2 is started by the image processing engine 14 executing a program when a still image is captured.

  Step # 101: The image processing engine 14 causes the imaging unit 13 to capture a recording still image in response to a user's shooting instruction. As a result, the image processing engine 14 acquires a color image to be processed from the imaging unit 13. Note that the color image at the step # 101 is a RAW image before color interpolation, and the signal value of each pixel corresponds to one of RGB components.

  Step # 102: The image processing engine 14 performs known white balance adjustment and color interpolation (Bayer interpolation) on the acquired color image. Note that all the RGB components are arranged in each pixel of the color image by the color interpolation of # 102.

  Step # 103: The image processing engine 14 designates a pixel of interest for brightness adjustment in the color image. Here, the image processing engine 14 sequentially designates all pixels of the color image as the target pixel. In addition, when changing the target pixel, the image processing engine 14 sequentially specifies the target pixel from left to right one line at a time starting from the upper left corner of the image.

  Step # 104: The image processing engine 14 performs a first gradation conversion process using a standard sRGB gamma gradation conversion curve G1 (equivalent to 1 / 2.2) on the pixel value of the target pixel. FIG. 3 shows a gradation conversion curve G1 applied in the first gradation conversion process. The horizontal axis in FIG. 3 is the input pixel value (output of the image sensor) before gradation conversion, and the vertical axis in FIG. 3 is the output pixel value after gradation conversion.

  Step # 105: The signal converter 21 converts the color space of the pixel of interest from RGB to YCbCr. In the process of # 105, the luminance signal Y is generated from the RGB signal values after gradation conversion is performed with G1 while maintaining the color space (sensor RGB) of the image sensor. For this reason, in the above-described processing, there is no occurrence of a gradation step in the high saturation portion, which is caused by matrix conversion from sensor RGB to sRGB.

For example, the signal conversion unit 21 in # 105 may obtain the YCbCr signal value from the RGB signal value of the target pixel by the following equations (1) to (3).
Y = 0.299 * R + 0.587 * G + 0.114 * B (1)
Cb = -0.194 * R-0.709 * G + 0.903 * B (2)
Cr = 0.872 * R-0.809 * G-0.063 * B (3)
In the above formula (1), Y is ITU-R BT. Conversion is performed using the coefficient defined in 601. In addition, for Cb and Cr in the above formulas (2) and (3), coefficients optimized using color charts are applied.

  By the way, since the luminance signal Y is generated from the sensors RGB, the luminance signal Y is different from the luminance to be generated by the original display system sRGB. The divergence increases as the saturation portion increases. In particular, in a color region such as blue or magenta, the luminance may increase too much. In addition, there is a problem that the frequency of color saturation increases due to the increase in luminance. Therefore, in step # 107, which will be described later, the luminance signal Y is corrected according to the hue and saturation.

  Step # 106: The luminance adjusting unit 22 divides the hue into a plurality of color areas based on the ratio of the color difference signals (Cb, Cr) of the target pixel, and the value of the color difference signal (Cb, Cr) for each divided color area. The gain used for brightness adjustment is determined according to the above. The luminance adjusting unit 22 in # 106 maps the color difference signal (Cb, Cr) of the target pixel to the CbCr color difference plane, and determines the gain from the color region to which the corresponding coordinate of the target pixel belongs on the color difference plane.

(About color area)
Here, the luminance adjustment unit 22 divides the color difference plane into six color regions for each hue, and associates a coefficient for determining a gain with each color region in advance. For example, the luminance adjustment unit 22 divides the color difference plane into six color regions in the rotation direction with reference to the origin (Cb, Cr = 0) of the color difference plane of CbCr.

FIG. 4 is a diagram illustrating an example of dividing the color difference plane. In FIG. 4, a red-yellow color region (hue = 0), a yellow-green color region (hue = 1), and a green-cyan color region (hue =) by three straight lines passing through the origin of the color difference plane. 2) The color difference plane is divided into six in a cyan-blue color area (hue = 3), a blue-magenta color area (hue = 4), and a magenta-red color area (hue = 5). The hexagonal range in FIG. 4 indicates the locus of the maximum saturation that CbCr can take.

In the YCbCr color space, the coordinates of the vertices of red (R), yellow (Y), green (G), cyan (C), blue (B), and magenta (M) are as follows.
R (1,0,0): (y: 0.29891; cb: -0.16874; cr: 0.50000)
Y (1,1,0): (y: 0.88552; cb: -0.50000; cr: 0.08131)
G (0,1,0): (y: 0.58661; cb: -0.33126; cr: -0.41869)
C (0,1,1): (y: 0.70109; cb: 0.16874; cr: -0.50000)
B (0,0,1): (y: 0.11448; cb: 0.50000; cr: -0.08131)
M (1,0,1): (y: 0.41339; cb: 0.33126; cr: 0.41869)
In addition, the slopes k of the three straight lines (straight line RC, straight line YB, and straight line GM) that divide the CbCr color difference plane shown in FIG. 4 are as follows.
k (RC) =-2.96314
k (YB) =-0.16262
k (GM) = 1.26393
Note that the luminance adjusting unit 22 in # 106 belongs to any color region in which the coordinates of the target pixel mapped to the color difference plane belong to hue = 0 to 5 using the conditional expression that divides the color difference plane by the straight line. What is necessary is just to determine. As described above, when the color difference plane is divided into a plurality of color areas by a straight line passing through the origin of the color difference plane, the amount of calculation when determining the color area to which the target pixel belongs is small.

(About brightness adjustment gain)
The luminance adjusting unit 22 determines the gain based on the above-described hue value (hue = 0 to 5) and the color difference signal (Cb, Cr). The gain is obtained by the following equation (4).
gain = 1.0- (KgainCb [hue] * Cb + KgainCr [hue] * Cr)… (4)
“KgainCb [hue]” in the above equation (4) is an array that returns a coefficient value of KgainCb by designating the color area hue to which the pixel of interest belongs, and “KgainCr [hue]” belongs to the pixel of interest. This is an array that returns the coefficient value of KgainCr by specifying the color area hue. Note that the coefficients of KgainCb and KgainCr are set so that the gains at the boundaries coincide with each other so that the gains do not become discontinuous at the hue boundaries. The value is set so as to approach the target luminance for various color samples. The luminance for each target color is not limited to one type, and a plurality of types of these coefficients may be prepared for each of a plurality of color tones such as “accurate color tone”, “preferable color tone”, and “flashy color tone”.

An example of the gain obtained by the method described above will be shown. The following gains are examples of gains at the vertices of the hexagon shown in FIG.
Red (R): 0.9
Yellow (Y): 1.0
Green (G): 0.9
Cyan (C): 0.5
Blue (B): 0.5
Magenta (M): 0.7
In FIG. 4, the vicinity of the origin is an achromatic region, and no luminance problem occurs in this portion. Then, when the gain is obtained with respect to this portion by the above-described equation (4), the gain is always about 1.0, and the luminance does not change by luminance adjustment described later. In the above example, the gains are all 1.0 or less, but a gain exceeding 1.0 may be used. Such a gain is a gain whose luminance is increased by luminance adjustment described later.

Step # 107: The luminance adjusting unit 22 adjusts the luminance signal of the pixel of interest using the following equation (5) using the gain determined in # 106.
Y = gain * Y… (5)
Step # 108: The image processing engine 14 performs a second gradation conversion process using the gradation conversion curve G2 on the luminance signal Y after the luminance adjustment of the target pixel. Note that a gradation conversion curve G2 applied in the second gradation conversion process is shown in FIG. The horizontal axis in FIG. 5 is an input pixel value before gradation conversion, and the vertical axis in FIG. 5 is an output pixel value after gradation conversion. This gradation conversion curve G2 is an S-shaped curve that reduces the gradation change in the low luminance portion and the high luminance portion. In the second gradation conversion process, contrast enhancement of the image and black tightening that compresses the gradation of the low luminance part are performed.

  Step # 109: The image processing engine 14 determines whether or not the processing of all the pixels of the color image has been completed. If the above requirement is satisfied (YES side), the process proceeds to # 110. On the other hand, if the above requirement is not satisfied (NO side), the process returns to # 103 and the image processing engine 14 repeats the above operation.

  Step # 110: The image processing engine 14 performs ring tone enhancement processing, noise removal processing, JPEG compression processing, etc. on the color image as necessary, and then stores the color image data in the storage medium 20 via the recording I / F 17. Record. Above, description of the flowchart of FIG. 2 is complete | finished.

  The electronic camera according to the first embodiment acquires a color image signal output from an image sensor, converts the acquired color image signal into a luminance signal and a color difference signal, and at the same time, saturation or hue of a target pixel of the color image. Accordingly, the luminance signal of the target pixel is adjusted.

  Thereby, in 1st Embodiment, generation | occurrence | production of the gradation level | step difference in a high saturation part can be suppressed. Therefore, it is possible to realize suitable color reproducibility without changing saturation and hue in signal conversion of a color image.

  Further, in the first embodiment, by adjusting the luminance signal, color reproducibility having a suitable gradation can be realized without lowering the exposure in order to avoid the minus area and the overflow area as in the prior art. Can do.

<Description of Second Embodiment>
The second embodiment is a modification of the first embodiment. Since the configuration of the electronic camera of the second embodiment is the same as that of the first embodiment, the same reference numerals are given and redundant description is omitted. In addition, regarding the operation example of the second embodiment, on the premise of the description of the first embodiment and FIG. 2, only the parts different from the first embodiment will be described, and the overlapping description with the first embodiment will be omitted.

  In the second embodiment, an example will be described in which saturation is adjusted by normalizing Cb and Cr of a pixel of interest with Y. This is to cope with a case where the luminance correction amount by the gain used in the luminance adjustment described in the first embodiment is reduced due to a decrease in the value of the color difference signal (Cb, Cr) in the dark portion.

  Instead of # 105 in FIG. 2 of the first embodiment, after obtaining the YCbCr signal value from the RGB signal value of the target pixel, the color difference signals (Cb, Cr) are normalized by the luminance signal Y, respectively. Then, instead of FIG. 4 described in # 106 of FIG. 2 of the first embodiment, the gain is determined using FIG. 6 and Expression (6) described later.

  FIG. 6 is another diagram showing an example of division of the color difference plane. Unlike FIG. 4, the horizontal axis of FIG. 6 is Cb / Y, and the vertical axis of FIG. 6 is Cr / Y. FIG. 6 shows six triangles shown in FIG. 6 as well as three straight lines (straight line RC, straight line YB, straight line GM) that divide the color difference plane of CbCr shown in FIG. It is divided into areas.

The gain is determined based on the value of hue (hue = 0 to 5) obtained based on FIG. 6 and the value of the color difference signal (Cb, Cr) normalized by the luminance signal Y. The gain is obtained by the following equation (6).
gain = 1.0- (KgainCby [hue] * Cb / Y + KgainCry [hue] * Cr / Y)… (6)
In the above equation (6), “KgainCby [hue]” is an array that returns the coefficient value of KgainCby by designating the color area hue to which the pixel of interest belongs, and “KgainCry [hue]” belongs to the pixel of interest. This is an array that returns a coefficient value of KgainCry by designating a color area hue. Note that the coefficients of KgainCby and KgainCry are set so that the gains at the boundaries coincide with each other so that the gains do not become discontinuous at the boundaries between the hues. The value is set so as to approach the target luminance for various color samples. The luminance for each target color is not limited to one type, and a plurality of types of these coefficients may be prepared for each of a plurality of color tones such as “accurate color tone”, “preferable color tone”, and “flashy color tone”.

An example of the gain calculated | required with the method mentioned above is shown. The following gains are examples of gains at the three vertexes of the triangle shown in FIG. 6 and the intersections of the three straight lines and each side of the triangle described above.
Red (R): 0.9
Yellow (Y): 1.0
Green (G): 0.9
Cyan (C): 0.9
Blue (B): 0.7
Magenta (M): 0.8
In FIG. 6 as well, when the gain is obtained from the above-described equation (6) for the achromatic region, the gain is always about 1.0, and the luminance does not change by the luminance adjustment described later.

  Thereafter, the same processing as # 107 to # 110 in FIG. 2 of the first embodiment is performed.

  In the case of the second embodiment, in addition to the operational effects of the first embodiment, the following effects can be obtained. In general, the value of CbCr is small in the dark portion (low luminance portion) of the image, and the correction amount in luminance adjustment is often small. However, in the case of the second embodiment, it is possible to realize suitable color reproducibility regardless of luminance by standardizing CbCr with Y.

<Description of Third Embodiment>
FIG. 7 is a diagram illustrating a configuration example of the image processing apparatus according to the third embodiment. The image processing apparatus according to the third embodiment is a computer in which a program for performing brightness adjustment processing on an input image is installed.

  A computer 31 illustrated in FIG. 7 includes a data reading unit 32, a storage device 33, a CPU 34, a memory 35, an input / output I / F 36, and a bus 37. The data reading unit 32, the storage device 33, the CPU 34, the memory 35, and the input / output I / F 36 are connected to each other via a bus 37. Furthermore, an input device 38 (keyboard, pointing device, etc.) and a monitor 39 are connected to the computer 31 via an input / output I / F 36. The input / output I / F 36 accepts various inputs from the input device 38 and outputs display data to the monitor 39.

  The data reading unit 32 is used when reading data of a color image to be processed (RAW image before color interpolation) or a program from the outside. The data reading unit 32 communicates with, for example, a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device based on a known communication standard. A communication device to perform (USB interface, LAN module, wireless LAN module, etc.).

  The storage device 33 is a storage medium such as a hard disk or a non-volatile semiconductor memory. The storage device 33 stores the above program and various data necessary for executing the program. The storage device 33 can also store image data read from the data reading unit 32.

  The CPU 34 is a processor that comprehensively controls each unit of the computer 31. The CPU 34 functions as the image processing engine 14, the signal conversion unit 21, and the luminance adjustment unit 22 of the first embodiment or the second embodiment by executing a program.

  The memory 35 temporarily stores various calculation results in the program. The memory 35 is, for example, a volatile SDRAM.

  Here, in the image processing apparatus according to the third embodiment, the CPU 34 acquires a color image to be processed from the data reading unit 32 or the storage device 33. Then, the CPU 34 executes the processes of # 102 to # 109 of the first embodiment or the second embodiment. Also in the third embodiment, substantially the same effects as those of the first embodiment and the second embodiment can be obtained.

<Supplementary items of the embodiment>
(Supplement 1): In the above embodiment, an example in which subject extraction is performed using YCbCr data of an image has been described. However, the present invention is not limited to the YCbCr color space, and can be applied to other color spaces (L * a * b * and the like).

  (Supplement 2): In the above embodiment, the example in which the color difference plane is divided into six color regions has been described. However, in the present invention, the number of divisions of the color difference plane may be changed as appropriate. For example, the color difference plane may be divided into three color regions using RGB coordinates as color region boundaries.

(Supplement 3): In the above embodiment, the example in which each function of the image processing apparatus is realized by software by a program has been described. However, in the image processing apparatus of the present invention, each function of the signal conversion unit 21 and the luminance adjustment unit 22 is set to, for example, an ASIC (Application Specific Integrated Circuit).
It may be realized in hardware using

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Imaging | photography optical system, 13 ... Imaging part, 14 ... Image processing engine, 15 ... 1st memory, 16 ... 2nd memory, 17 ... Recording I / F, 18 ... Display part, 19 ... Operation part , 20 ... storage medium, 21 ... signal conversion unit, 22 ... luminance adjustment unit, 31 ... computer, 32 ... data reading unit, 33 ... storage device, 34 ... CPU, 35 ... memory, 36 ... input / output I / F, 37 ... Bus, 38 ... Input device, 39 ... Monitor

Claims (6)

An acquisition unit for acquiring a color image signal output from the image sensor;
A signal conversion unit for converting the signal of the color image into a luminance signal,
A luminance adjustment unit that adjusts the luminance signal of the pixel of interest according to the saturation and hue of the pixel of interest of the color image,
The luminance processing unit is an image processing apparatus that obtains an adjustment amount of the luminance signal so that the luminance decreases as the saturation of the pixel of interest increases. The image processing apparatus according to claim 1 .
The signal conversion unit converts the color image signal into a luminance signal and a color difference signal,
The luminance adjustment unit divides the color difference plane into at least three color regions according to hues, associates different coefficients with each color region, maps the color difference signal of the pixel of interest of the color image onto the color difference plane, and An image processing apparatus that obtains the adjustment amount using a coefficient of a color region to which a signal belongs. The image processing apparatus according to claim 2 ,
The luminance adjustment unit is an image processing apparatus that obtains the adjustment amount by normalizing the value of the color difference signal with the value of the luminance signal. The image processing apparatus according to claim 3 .
The luminance adjusting unit divides a color difference plane based on the normalized color difference signal into at least three color regions for each hue, associates different coefficients with the color regions, and outputs a color difference signal of a target pixel of the color image. An image processing apparatus that maps to a color difference plane and obtains the adjustment amount using a coefficient of a color region to which the color difference signal belongs. An imaging unit for imaging a color image;
An imaging device comprising: the image processing device according to any one of claims 1 to 4 . Processing to obtain a color image signal output from the image sensor;
A process of converting a signal of the color image into a luminance signal,
A process of adjusting the luminance signal of the target pixel according to the saturation and hue of the target pixel of the color image,
In the process of adjusting the luminance signal, a program for performing a process of obtaining an adjustment amount of the luminance signal so that the luminance decreases as the saturation of the pixel of interest increases.

Images