JP2018107659A - Luminance signal generating device, luminance signal generating method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve quality of a luminance signal.SOLUTION: A luminance signal generating circuit 210 determines a synthesis ratio of an OG signal and a SWY signal in accordance with a color determination signal DC, and generates a base luminance signal by synthesizing the OG signal and the SWY signal in accordance with the determined synthesis ratio. The luminance signal generating circuit 210 determines the synthesis ratio of the OG signal and the SWY signal in accordance with a red color determination signal Dr, a low frequency determination signal Dl, and the edge determination signal De, and generates a luminance signal for a high range by synthesizing the OG signal and the SWY signal in accordance with the determined synthesis ratio.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a luminance signal generation device, a luminance signal generation method, and a program, and is particularly suitable for use in generating a luminance signal from a color signal output from an image sensor.

  In general, when a color image is generated using an imaging device capable of detecting the amount of light, such as a CCD image sensor or a CMOS image sensor, light transmitted through a color filter is incident on an imaging device. There are various types of color filters depending on the type of color and the arrangement of colors assigned to each pixel. As color types, primary colors (red, green, blue) or complementary colors (cyan, magenta, yellow) are widely used. As the color arrangement, the Bayer arrangement is widely used.

FIG. 13 is a diagram showing one unit of the primary color Bayer array. In FIG. 13, R is red, G1 and G2 are green, and B is blue. Actually, the same arrangement as that shown in FIG. 13 is repeated according to the number of pixels of the image sensor.
Two methods are known as methods for generating a luminance signal in accordance with a color signal output from an image sensor.

In the first method, when the luminance signal is generated using the primary color Bayer array color filter shown in FIG. 13, the signals of red, green, and blue are processed independently. This is referred to as the “Out Of Green” method. The Out Of Green method is a method for creating a luminance signal with a green (G) signal as a center. For example, when processing a green signal, the luminance signal generation device inserts 0 (zero) as a pixel value into pixels other than the pixel corresponding to green in the RAW signal obtained by digitizing the color signal output from the image sensor. . Then, the luminance signal generation device performs a low-pass filter (V-LPF) process for limiting a vertical band and a low-pass filter for limiting a horizontal band for a RAW signal in which 0 (zero) is inserted as a pixel value. (H-LPF) processing is performed. The luminance signal generation apparatus similarly processes the red signal and the blue signal so that each pixel has all the signals of the red signal, the green signal, and the blue signal. Then, the luminance signal generation device obtains the luminance Y from the green signal, red signal, and blue signal created as described above using, for example, the following equation (1).
Y = 0.3R + 0.59G + 0.11B (1)

Further, the luminance signal generation device generates a high frequency emphasis signal from only the green signal, and adds the generated high frequency emphasis signal to the luminance Y (hereinafter referred to as OG signal).
Note that the luminance signal generation device may use a green signal subjected to low-pass filter processing as the luminance Y instead of giving all the signals of the red signal, the green signal, and the blue signal to each pixel. In this case, the OG signal is generated only from the green signal.

  On the other hand, in the second method, when the luminance signal is generated using the primary color Bayer array color filter shown in FIG. 13, all the red (R), green (G), and blue (B) pixels are used for luminance. Create a signal. This is called a SWY system. In the SWY system, a RAW signal obtained by digitizing a color signal output from an image sensor is regarded as a luminance signal as it is without being distinguished by color.

  FIG. 14 is a diagram showing a concept of a luminance signal obtained by the SWY method. In FIG. 14, the suffix (subscript) added to the luminance Y indicates the position of the pixel. Usually, in order to suppress the generation of carriers due to sampling, the result of applying a low-pass filter (LPF) whose output is 0 (zero) at the Nyquist frequency in the horizontal and vertical directions is used as the base signal. For example, if the filter coefficient of the LPF is [1 2 1] in both the horizontal direction and the vertical direction, the luminance signal generation device can obtain the output Y22 from the LPF corresponding to the pixel of Y22 by the following equation (2). it can.

Y22 = (Y11 + 2 × Y12 + Y13 + 2 × Y21 + 4 × Y22 + 2 × Y23 + Y31 + 2 × Y32 + Y33) / 16 (2)
Furthermore, by generating a high frequency emphasis signal from the base signal and adding the original base signal to the high frequency emphasis signal, it is possible to obtain a signal in which the high frequency is compensated (hereinafter referred to as SWY signal). .

  FIG. 15 is a diagram illustrating spatial frequency characteristics that can be resolved in the OG signal and the SWY signal. In FIG. 15, the x-axis represents the frequency space in the horizontal (H) direction of the subject, and the y-axis represents the frequency space in the vertical (V) direction. FIG. 15 shows that the spatial frequency increases as the distance from the origin increases.

  In the Out Of Green method, a luminance signal is generated centering on a green (G) signal. For this reason, the spatial frequency of the OG signal that can be resolved in the horizontal and vertical directions is equal to the Nyquist frequency (π / 2 on the axis in FIG. 15). However, there is a line in which no pixel exists in the oblique direction. For this reason, the limit spatial frequency in the OG signal that can be resolved in the oblique direction is lower than that in the horizontal direction and the vertical direction. As a result, the inside of the rhombus region 1500 shown in FIG. 15 is resolved by the OG signal. Possible spatial frequency.

  On the other hand, in the SWY method, a luminance signal is generated using all pixels. For this reason, when the subject is achromatic, the square area 1501 shown in FIG. 15 has a spatial frequency that can be resolved by the SWY signal. However, for example, in a red subject, a luminance signal is hardly output from pixels other than the pixel corresponding to red (R). For this reason, in the case of a red subject, only a square region 1502 in a quarter range has a spatial frequency that can be resolved by the SWY method, compared to an achromatic subject.

  As described above, both the Out Of Green method and the SWY method have a disadvantage regarding a resolvable spatial frequency. Therefore, in Patent Document 1, it is determined whether the image is a monochrome image or a color image, and only when the image is an achromatic subject, the SWY signal is applied to the oblique regions 1504a to 1504d of the OG signal shown in FIG. A technique for generating a luminance signal subjected to the replacement in (1) is disclosed.

  As shown in Expression (1) and Expression (2), the weights of the green signal, red signal, and blue signal for the generated luminance signal differ between the Out Of Green method and the SWY method. However, when the subject is an achromatic color, the levels of the green signal, the red signal, and the blue signal are the same, and therefore the level of the luminance signal does not change even if the OG signal is replaced with the SWY signal. Further, in the technique described in Patent Document 1, a high-frequency signal is extracted by applying a bandpass filter to the generated luminance signal, and the extracted high-frequency signal is added as a luminance signal based on the generated luminance signal. The final luminance signal is generated by indenting.

  Patent Document 2 discloses the following technique. First, a base luminance signal is a luminance signal generated by the Out Of Green method. The high frequency signal generated from the OG signal and the high frequency signal generated from the SWY signal are combined according to the determination result of whether or not the subject is a red (blue) subject. Then, a final luminance signal is generated by adding the synthesized high frequency signal to the luminance signal as a base.

  In Patent Document 2, as in the case of a red (blue) subject, the spatial frequency that can be resolved by the SWY signal is a square region 1502, which is narrower than the diamond region 1500 that is the spatial frequency that can be resolved by the OG signal. Increase the use ratio of the high frequency signal generated from the OG signal. On the other hand, if not, the usage rate of the high frequency signal is increased from the SWY signal. At this time, since the base luminance signal is a luminance signal generated by the Out Of Green method, the level of the luminance signal does not change even for a chromatic subject.

JP 2009-105977 A JP 2008-72377 A

However, with the technique described in Patent Document 1, a high-frequency signal generated from the OG signal is used for subjects other than achromatic subjects. For this reason, in the technique described in Patent Document 1, the spatial frequency at which a high-frequency signal can be resolved for a chromatic color object other than a red (blue) object is narrower than the technique described in Patent Document 2. In the technique described in Patent Document 2, the base luminance signal is a luminance signal generated by the Out Of Green method. For this reason, in the technique described in Patent Document 2, the spatial frequency at which the luminance signal serving as a base for an achromatic object can be resolved is narrower than the technique described in Patent Document 1. As described above, in the techniques described in Patent Documents 1 and 2, there is a possibility that the quality of the luminance signal is deteriorated.
The present invention has been made in view of such problems, and an object thereof is to improve the quality of a luminance signal.

  The luminance signal generation apparatus of the present invention is an image signal captured by an imaging unit having a color filter and an image sensor, and generates a luminance signal from an image signal composed of a plurality of color signals including the first color signal. A luminance signal generation device that performs the weighting of only the first color signal or the weight of the first color signal among the plurality of color signals in the image signal. First generation means for generating a first luminance signal using the plurality of color signals weighted so that the color signal that has passed through the color filter region corresponding to each pixel of the image sensor Based on first information derived from the image signal, second generation means for generating a second luminance signal using the second luminance signal as the luminance signal of the first luminance signal, and the second luminance signal, Determine the composition ratio of Derived from the image signal; and third generation means for generating a third luminance signal based on the determined combination ratio and at least one of the first luminance signal and the second luminance signal. Based on information different from the first information as the second information, a synthesis ratio of the first luminance signal and the second luminance signal is determined, and the determined synthesis ratio and the first information And fourth generation means for generating a fourth luminance signal that is a higher frequency luminance signal than the third luminance signal based on at least one of the luminance signal and the second luminance signal. It is characterized by.

  According to the present invention, the quality of the luminance signal can be improved.

It is a figure which shows the structure of an imaging device. It is a figure which shows the structure of an image process part. It is a figure which shows the structure of a luminance signal generation circuit. It is a flowchart which shows the process of a chromatic color determination part. It is a figure which shows the relationship between a 1st chroma signal and a chromatic color determination signal. It is a flowchart which shows the process of a red determination part. It is a figure which shows the relationship between a 2nd saturation signal and a red determination signal. It is a figure which shows the structure of a low frequency area | region determination part. It is a flowchart which shows the process of a low frequency area | region determination part. It is a figure which shows an example of the relationship between the signal for low frequency detection, and the low frequency determination signal. It is a flowchart which shows the process of a luminance signal synthetic | combination process part. It is a flowchart which shows the process of the luminance signal synthetic | combination process part for high regions. It is a figure which shows 1 unit of a primary color Bayer arrangement | sequence. It is a figure which shows the concept of the luminance signal Y obtained by SWY system. It is a figure which shows the range which can be resolved in OG signal and SWY signal.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the luminance generation apparatus is applied to an imaging apparatus will be described as an example.
FIG. 1 is a block diagram illustrating an example of the configuration of the imaging apparatus 100.
The lens group 101 is a zoom lens including a focus lens. The shutter 102 has an aperture function. The shutter 102 exposes the image sensor included in the imaging unit 103 in accordance with the control of the system control unit 50. The imaging unit 103 includes an imaging element such as a CCD image sensor or a CMOS image sensor. The imaging element converts an optical image obtained through the lens group 101 into an electrical signal by photoelectric conversion. In the present embodiment, the imaging unit 103 includes a primary color Bayer array color filter. The A / D conversion unit 104 converts the analog signal read from the imaging unit 103 into a digital signal and outputs the image signal to the image processing unit 105.

  The image processing unit 105 includes a signal processing circuit that performs signal processing such as so-called development processing on the image signal output from the A / D conversion unit 104 or the image signal output from the memory control unit 107. The image processing unit 105 performs various types of image processing such as white balance adjustment, luminance signal generation, and gamma correction.

  The image memory 106 temporarily stores an image signal when the image processing unit 105 performs various image processing. The image memory 106 stores an image signal read from the recording medium 112 via the recording medium interface (I / F) 111 and an image signal to be displayed on the display unit 109. A memory control unit 107 controls reading and writing of the image memory 106. The D / A converter 108 converts the digital signal into an analog signal. For example, the D / A converter 108 converts image display data stored in the image memory 106 into an analog signal and outputs the analog signal to the display unit 109.

  The display unit 109 includes a display device such as a liquid crystal display. The display unit 109 displays an image captured by the imaging apparatus 100, an image read from the recording medium 112, a live view image, and the like. The display unit 109 displays a user interface for the user to perform an operation on the imaging apparatus 100. The codec unit 110 compresses and decodes the image signal. The codec unit 110 encodes or decodes the image signal recorded in the image memory 106 in a format compliant with a standard such as MPEG.

  The recording medium I / F 111 mechanically and electrically connects the recording medium 112 to the imaging device 100. The recording medium 112 is a detachable recording medium such as a semiconductor memory card or a card-type hard disk. The system control unit 50 includes a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The system control unit 50 executes the program stored in the non-volatile memory 121 by developing it in the work area of the system memory 122 and controls each function of the entire imaging apparatus 100. For example, the system control unit 50 performs exposure control and distance measurement control based on the result of processing by the image processing unit 105 and the like. As a result, the system control unit 50 performs TTL (through the lens) AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like.

  The operation unit 120 includes a touch panel as an interface displayed on the display unit 109 described above, buttons, and switches. The operation unit 120 notifies the system control unit 50 of the content of the user operation on the operation unit 120. The non-volatile memory 121 includes a non-volatile semiconductor memory that stores programs, parameters, and the like as an auxiliary storage device. Specifically, the nonvolatile memory 121 includes, for example, an EEPROM. The system memory 122 is a main storage device, and develops a program read from the nonvolatile memory 121 and stores constants and variables for operation of the system control unit 50.

FIG. 2 is a block diagram illustrating an example of the configuration of the image processing unit 105.
The image signal generation circuit 201 inputs the image signal output from the A / D conversion unit 104. In the present embodiment, it is assumed that the image signal is an RGB image signal configured by a primary color Bayer array.

The WB circuit 202 calculates a WB correction value based on the information of the image signal output from the image signal generation circuit 201. The WB circuit 202 corrects the white balance of the image signal using the calculated WB correction value.
A color conversion matrix (MTX) circuit 203 multiplies the image signal by a color gain so that the image signal whose white balance has been corrected by the WB circuit 202 is reproduced in an optimal color, and the image signal is converted into two color difference signals. Convert to
A low-pass filter (LPF) circuit 204 limits the band of the color difference signal. A CSUP (Chroma Suppress) circuit 205 suppresses a false color signal in a saturated portion of the color difference signal band-limited by the LPF circuit 204.

The image signal whose white balance has been corrected by the WB circuit 202 is also supplied to the luminance signal generation circuit 210. The luminance signal generation circuit 210 generates a luminance signal based on the image signal whose white balance has been corrected by the WB circuit 202. Further, the luminance signal generation circuit 210 generates a luminance signal to which edge enhancement processing is applied.
The color difference signal output from the CSUP circuit 205 and the luminance signal output from the luminance signal generation circuit 210 are converted into RGB signals by the RGB conversion circuit 206.
A gamma (γ) correction circuit 207 performs gamma correction (gradation correction) on the RGB signal in accordance with a predetermined gamma characteristic. The gamma-corrected RGB signal is converted into a YUV signal by the color luminance conversion circuit 208, and then compressed and encoded by the JPEG compression circuit 209, and recorded as an image data file in the image memory 106 or the like.

FIG. 3 is a block diagram illustrating an example of the configuration of the luminance signal generation circuit 210. In the following description, an image signal whose white balance has been corrected by the WB circuit 202 is referred to as a RAW signal as necessary.
The RAW signal 300 is input to the OG signal generation unit 302, the SWY signal generation unit 303, the chromatic color determination unit 304, the red determination unit 305, and the low frequency region determination unit 306, respectively.
The OG signal generation unit 302 generates an OG signal from the RAW signal 300. The OG signal is input to the luminance signal synthesis processing unit 307 and the high frequency luminance signal synthesis processing unit 308, respectively. The OG signal input to the luminance signal synthesis processing unit 307 is a luminance signal obtained by using Equation (1) from the green signal, the red signal, and the blue signal, and is input to the high frequency luminance signal synthesis processing unit 308. The OG signal is a luminance signal generated from only the green signal. Although it is not always necessary to do this, it is preferable to do so because it is possible to suppress the moiré signal from being emphasized by the processing by the high frequency luminance signal synthesis processing unit 308.

  In the present embodiment, for example, by using the RAW signal 300, an example of an image signal including a plurality of color signals including the first color signal is realized. Further, for example, by using the OG signal generation unit 302, an example of the first generation unit is realized. Further, for example, an example of the first luminance signal is realized by using the OG signal. Further, for example, the first color signal is realized by using the green signal.

The SWY signal generation unit 303 generates a SWY signal from the RAW signal 300. The SWY signal is input to the luminance signal synthesis processing unit 307 and the high frequency luminance signal synthesis processing unit 308, respectively.
In the present embodiment, an example of the second generation unit is realized by using the SWY signal generation unit 303. For example, an example of the second luminance signal is realized by using the SWY signal.

The chromatic color determination unit 304 determines whether or not the subject has a chromatic color based on the RAW signal 300. Here, an example of processing in the chromatic color determination unit 304 will be described with reference to the flowchart of FIG. Note that the flowchart of FIG. 4 is executed for each pixel. First, the chromatic color determination unit 304 calculates a first saturation signal C1 from the RAW signal 300 according to the following equations (3) to (6) (step S401). R, G, and B are the values of the red signal, the green signal, and the blue signal, respectively.
C1 = | Cr | + | Cb | (3)
Y = 0.3R + 0.59G + 0.11B (4)
Cr = 0.713 (R−Y) (5)
Cb = 0.564 (B−Y) (6)

  Next, the chromatic color determination unit 304 determines whether or not the first saturation signal C1 is less than the threshold value Th1 (step S402). As a result of this determination, when the first saturation signal C1 is less than the threshold Th1 (YES in step S402), the chromatic color determination unit 304 determines that the pixel to be processed is an achromatic color. In this case, the chromatic color determination unit 304 sets 1.0 as the chromatic color determination signal Dc (step S403). On the other hand, if the first saturation signal C1 is greater than or equal to the threshold Th1 (NO in step S402), the chromatic color determination unit 304 determines whether or not the first saturation signal C1 exceeds the threshold Th2. (Step S404). The threshold value Th2 is a value that exceeds the threshold value Th1. As a result of this determination, if the first saturation signal C1 exceeds the threshold Th2 (YES in step S404), the chromatic color determination unit 304 sets 0.0 as the chromatic color determination signal Dc (step S405). .

  On the other hand, when the first saturation signal C1 is between the threshold Th1 and the threshold Th2 (NO in step S404), the chromatic color determination unit 304 uses 0.0 to 1.0 as the chromatic color determination signal Dc. A numerical value between is set (step S406). Specifically, the chromatic color determination unit 304 linearly takes a numerical value between 0.0 and 1.0 for the chromatic color determination signal Dc according to the distance from the thresholds Th1 and Th2 of the first saturation signal C1. Like that. FIG. 5 is a diagram illustrating an example of the relationship between the first saturation signal C1 and the chromatic color determination signal Dc. When the first saturation signal C1 is between the threshold values Th1 and Th2, the chromatic color determination signal Dc has a value corresponding to the first saturation signal C1 in the diagonal line shown in FIG. As shown in FIG. 5, this straight line is obtained when the chromatic color determination signal Dc is 1.0 and the first saturation signal C1 is the threshold value Th1, and the chromatic color determination signal Dc is 0.0. And a straight line connecting a point when the first saturation signal C1 is the threshold Th2.

Note that the chromatic color determination unit 304 may adjust a region for determining whether the color is a chromatic color. For example, the chromatic color determination unit 304 applies, for example, a 3 × 3 maximum value filter to the first saturation signal C1 to reduce the sensitivity of the region for determining whether or not the color is chromatic. Can do. The sensitivity may be lowered by using, for example, a low-pass filter instead of the maximum value filter. Further, the chromatic color determination unit 304 may normalize the first saturation signal C1 with brightness to detect a dark part. Normalization with brightness can be performed, for example, by dividing the first saturation signal C1 (saturation) by the luminance Y.
The chromaticity determination signal Dc output from the chromatic color determination unit 304 is input to the luminance signal synthesis processing unit 307.

  In the present embodiment, for example, an example of the first derivation unit is realized by using the chromatic color determination unit 304. Moreover, in this embodiment, an example of the 1st information based on an image signal is implement | achieved by using chromaticity determination signal Dc, for example. In the present embodiment, the chromaticity determination signal Dc is also an example of information regarding the first color in the image signal. Further, the chromaticity determination signal Dc is any one of a color in the image signal being a chromatic color, a color in the image signal being an achromatic color, and a degree of proximity of the color in the image signal to the achromatic color. It is also an example of information indicating that.

Returning to the description of FIG. 3, the red determination unit 305 determines whether or not the subject is red based on the RAW signal 300. Here, an example of processing in the red determination unit 305 will be described with reference to the flowchart of FIG. Note that the flowchart of FIG. 6 is executed for each pixel. First, the red determination unit 305 calculates a second saturation signal C2 from the RAW signal 300 according to the following equations (7) to (9) (step S601). Note that max (Cr, 0) indicates that the larger of Cr and 0 (zero) is adopted.
C2 = max (Cr, 0) (7)
Y = 0.3R + 0.59G + 0.11B (8)
Cr = 0.713 (R−Y) (9)

  Next, the red determination unit 305 determines whether or not the second saturation signal C2 is less than the threshold value Th3 (step S602). If the result of this determination is that the second saturation signal C2 is less than the threshold Th3 (YES in step S602), the red determination unit 305 does not indicate that the pixel value to be processed indicates a red subject. judge. In this case, the red determination unit 305 sets 1.0 as the red determination signal Dr (step S603). On the other hand, when the second saturation signal C2 is greater than or equal to the threshold Th3 (NO in step S602), the red determination unit 305 determines whether or not the second saturation signal C2 exceeds the threshold Th4 ( Step S604). The threshold value Th4 is a value that exceeds the threshold value Th3. As a result of the determination, when the second saturation signal C2 exceeds the threshold Th4 (YES in step S604), the red determination unit 305 sets 0.0 as the red determination signal Dr (step S605).

  On the other hand, when the second saturation signal C2 is between the threshold value Th3 and the threshold value Th4 (NO in step S604), the red determination unit 305 sets the red determination signal Dr between 0.0 and 1.0. Is set (step S606). Specifically, the red determination unit 305 is configured such that the red determination signal Dr takes a numerical value between 0.0 and 1.0 linearly according to the distance from the threshold values Th3 and Th4 of the second saturation signal C2. To do. FIG. 7 is a diagram illustrating an example of a relationship between the second saturation signal C2 and the red determination signal Dr.

  Note that the red determination unit 305 may adjust the region for determining whether or not the color is red, similarly to the chromatic color determination unit 304. That is, the red determination unit 305 applies, for example, a 3 × 3 maximum value filter to the second saturation signal C2 to reduce the sensitivity of the region for determining whether the color is red, The saturation signal C2 may be normalized by brightness to detect a dark part.

Further, the red determination unit 305 may calculate the second saturation signal C2 from the RAW signal 300 according to the following equations (10) to (13).
C2 = max (Cr, 0) + max (Cb, 0) (10)
Y = 0.3R + 0.59G + 0.11B (11)
Cr = 0.713 (R−Y) (12)
Cb = 0.564 (B−Y) (13)

By calculating the second saturation signal C2 according to the equations (10) to (13), it is possible to determine not only a red subject but also a blue subject. In this way, it is possible to determine a blue subject that hardly outputs a luminance signal from pixels other than the pixel corresponding to blue in the primary color filter. Further, it may be determined whether the subject is red or blue. Depending on the spectral sensitivity characteristics of the color filter, it may be selected between determining whether the subject is red or determining whether the subject is blue.
The red determination signal Dr output from the red determination unit 305 is input to the high frequency luminance signal synthesis processing unit 308.

  In the present embodiment, an example of the second deriving unit is realized by using the red determination unit 305. In the present embodiment, for example, by using the red determination signal Dr, an example of information that is at least partially different from the first information is realized as the second information based on the image signal. In the present embodiment, the red determination signal Dr is also an example of information regarding the second color in the image signal. The red determination signal Dr includes a color in the image signal that is the color of the second color signal, a color in the image signal that is different from the color of the second color signal, and the image signal. It is also an example of information indicating one of the degrees of proximity of the color of the color to the color of the second color signal.

  Returning to the description of FIG. 3, the low frequency region determination unit 306 determines whether or not the image is an image of a low frequency subject based on the RAW signal 300. The low frequency region determination unit 306 can detect a low frequency region by applying a filter for detecting the low frequency region to the RAW signal 300. FIG. 8 is a block diagram illustrating an example of the configuration of the low frequency region determination unit 306. FIG. 9 is a flowchart for explaining an example of processing of the low frequency region determination unit 306. Note that the flowchart of FIG. 9 is executed for each pixel.

  The low-pass filter (V-LPF) 800 in the vertical direction performs a filtering process on the RAW signal 300, and limits the frequency band of the RAW signal 300 in the vertical direction (step S901). Next, the horizontal low-pass filter (H-LPF) 801 performs a filtering process on the RAW signal in which the vertical high-frequency frequency band is limited in step S901, and the horizontal high-frequency frequency of the RAW signal is determined. The bandwidth is limited (step S902). Next, the horizontal band-pass filter (H-BPF) 802 performs filtering on the RAW signal in which the vertical and horizontal frequency bands are limited in steps S901 and S902, and performs horizontal processing on the RAW signal. Is limited (step S903). Next, the vertical band-pass filter (V-BPF) 803 performs filtering on the RAW signal in which the vertical and horizontal frequency bands are limited in steps S901 and S902, and performs vertical processing on the RAW signal. Is limited (step S904).

  Next, the adder 804 adds the signal output from the horizontal bandpass filter (H-BPF) 802 and the signal output from the vertical bandpass filter (V-BPF) 803 (step). S905). Next, the absolute value circuit (ABS) 805 takes the absolute value of the signal output from the adder 804, and outputs a signal indicating the absolute value as the low frequency detection signal LowF (step S906). Next, the low frequency determination processing unit 806 determines whether or not the low frequency detection signal LowF is less than the threshold value Th5 (step S907). As a result of this determination, when the low frequency detection signal LowF is less than the threshold Th5 (YES in step S907), the low frequency determination processing unit 806 determines that the pixel to be processed is not in the low frequency region. Then, the low frequency determination processing unit 806 sets 1.0 as the low frequency determination signal Dl (step S908). On the other hand, when the low frequency detection signal LowF is equal to or higher than the threshold Th5 (NO in step S907), the low frequency determination processing unit 806 determines whether or not the low frequency detection signal LowF exceeds the threshold Th6 ( Step S909). The threshold value Th6 is a value that exceeds the threshold value Th5. As a result of this determination, when the low frequency detection signal LowF exceeds the threshold Th6 (YES in step S909), the low frequency determination processing unit 806 determines that the pixel to be processed is a low frequency region. Then, the low frequency determination processing unit 806 sets 0.0 as the low frequency determination signal Dl (step S910).

  On the other hand, when the low frequency detection signal LowF is between the threshold value Th5 and the threshold value Th6 (NO in step S909), the low frequency determination processing unit 806 uses 0.0 to 1.0 as the low frequency determination signal Dl. A numerical value between is set (step S911). Specifically, the low frequency determination processing unit 806 linearly takes a numerical value between 0.0 and 1.0 for the low frequency determination signal Dl according to the distance from the thresholds Th5 and Th6 of the low frequency detection signal LowF. Like that. FIG. 10 is a diagram illustrating an example of the relationship between the low frequency detection signal LowF and the low frequency determination signal Dl. The low frequency determination signal Dl output from the low frequency region determination unit 306 is input to the high frequency luminance signal synthesis processing unit 308. In the present embodiment, for example, an example of the second information is realized by using the low frequency detection signal LowF. In addition, for example, the low frequency detection signal LowF is an example of information on the spatial frequency in the image signal.

  Note that the low frequency determination processing unit 806 may adjust the region for determining whether or not the low frequency region is the same as the chromatic color determination unit 304 and the red determination unit 305. For example, the low frequency determination processing unit 806 applies a 3 × 3 maximum value filter to the low frequency detection signal LowF to reduce the sensitivity of the region for determining whether or not it is a low frequency region, The low frequency detection signal LowF may be normalized by brightness to detect a dark part.

  Next, an example of processing in the luminance signal synthesis processing unit 307 will be described using the flowchart of FIG. Note that the flowchart of FIG. 11 is executed for each pixel. The luminance signal synthesis processing unit 307 synthesizes the OG signal output from the OG signal generation unit 302 and the SWY signal output from the SWY signal generation unit 303 using the chromatic color determination signal Dc obtained as described above. Then, a base luminance signal is generated.

  First, the luminance signal synthesis processing unit 307 determines whether or not the RAW signal 300 input to the luminance signal generation circuit 210 is an achromatic subject signal, that is, whether or not the chromatic color determination signal Dc is 1.0. Is determined (step S1101). As a result of this determination, when the RAW signal 300 is an achromatic subject signal (the chromatic color determination signal Dc is 1.0) (YES in step S1101), the luminance signal synthesis processing unit 307 outputs the base luminance signal. The SWY signal is selected (step S1102). On the other hand, when the RAW signal 300 is not an achromatic subject signal (the chromatic color determination signal Dc is 1.0), the luminance signal synthesis processing unit 307 determines whether or not the chromatic color determination signal Dc is 0.0. (Step S1103). As a result of this determination, when the chromatic color determination signal Dc is 0.0 (YES in step S1103), the luminance signal synthesis processing unit 307 selects the OG signal as the base luminance signal (step S1104).

When the chromatic color determination signal Dc becomes a value between 0.0 and 1.0 (NO in step S1103), the luminance signal synthesis processing unit 307 determines that the OG signal and the OG signal according to the following equation (14). A signal obtained by combining the SWY signal is generated as a base luminance signal (step S1105).
[Base luminance signal] = Dc × [SWY signal] + (1.0−Dc) × [OG signal] (14)
By generating the base luminance signal as shown in Expression (14), the switching of the base luminance signal can be made smooth.

  By combining the OG signal and the SWY signal by the above method, the SWY signal is used as a base luminance signal for an achromatic object whose spatial frequency that enables the resolution of the SWY signal becomes wider than the OG signal. It becomes possible to do. In addition, by limiting the subject using the SWY signal to an achromatic subject, the composition ratio of the green signal, the red signal, and the blue signal can be achieved even when the OG signal and the SWY signal are combined in accordance with the chromatic color determination signal Dc. It is possible to make it difficult to generate a difference in level of the luminance signal due to the difference.

  In the present embodiment, for example, by using the chromatic color determination unit 304 and the luminance signal synthesis processing unit 307, an example of a third generation unit is realized. In addition, for example, by using the coefficient (Dc, (1.0−Dc)) of the equation (14), an example of a combination ratio when generating the third luminance signal is realized. Further, for example, an example of the third luminance signal is realized by using the base luminance signal.

Next, an example of processing in the high frequency luminance signal synthesis processing unit 308 will be described with reference to the flowchart of FIG. Note that the flowchart of FIG. 12 is executed for each pixel. The high frequency luminance signal synthesis processing unit 308 uses the red determination signal Dr and the low frequency determination signal Dl obtained as described above, and outputs the OG signal and SWY signal generation unit 303 output from the OG signal generation unit 302. Are combined with the SWY signal output from the signal to generate a high-frequency luminance signal.
First, the high frequency luminance signal synthesis processing unit 308 calculates a synthesis rate α according to the following equation (15) based on the red color determination signal Dr and the low frequency determination signal Dl (step S1201).
α = Dr × Dl (15)

  That is, the composition rate α is 0.0 when the subject is a red subject or a low-frequency subject. Further, the composition ratio α takes a value of 1.0 when the subject is not a red subject and is not a low-frequency subject. The composition rate α is closer to 0.0 (smaller value) as the subject is closer to red, and closer to 0.0 (smaller value) as the subject has a lower spatial frequency.

  Next, the high frequency luminance signal synthesis processing unit 308 determines whether or not the synthesis rate α is 1.0 (step S1202). If the result of this determination is that the synthesis rate α is 1.0 (YES in step S1202), the high frequency luminance signal synthesis processing unit 308 selects the SWY signal as the high frequency luminance signal (step S1202). S1203). On the other hand, when the synthesis rate α is not 1.0 (NO in step S1202), the high frequency luminance signal synthesis processing unit 308 determines whether or not the synthesis rate α is 0.0 (step S1204). . If the result of this determination is that the synthesis rate α is 0.0 (YES in step S1204), the high frequency luminance signal synthesis processing unit 308 selects the OG signal as the high frequency luminance signal (step S1204). S1205).

When the synthesis rate α is a value between 0.0 and 1.0 (NO in step S1204), the high frequency luminance signal synthesis processing unit 308 generates an OG signal according to the following equation (16). And a signal obtained by synthesizing the SWY signal are generated as a high-frequency luminance signal (step S1206).
[Luminance signal for high frequency] = α × [SWY signal] + (1.0−α) × [OG signal] (16)
By generating the high frequency luminance signal as shown in Expression (16), the high frequency luminance signal can be switched smoothly.

  By synthesizing the OG signal and the SWY signal by the above method, the SWY signal is used for the high frequency band for the red subject whose spatial frequency that can be resolved is narrower than the OG signal. It can be used as a signal. In this case, only a high-frequency signal is extracted from the high-frequency luminance signal by finally applying a bandpass filter. For this reason, even when the OG signal and the SWY signal are combined, the difference in level of the luminance signal in a chromatic subject other than red due to the difference in the composition ratio of the green signal, the red signal, and the blue signal is unlikely to be a problem.

  Also, by referring to the low frequency determination signal Dl and using the OG signal in the low frequency region, it is possible to select the OG signal in the region of the spatial frequency that can be resolved by the OG signal. As a result, generally, the SWY signal having a high composition ratio with respect to the luminance signal of the B pixel and the R pixel, which causes degradation due to noise with respect to the G pixel due to white balance processing, is selected, and the luminance signal is degraded due to noise. It is possible to reduce this.

  In the present embodiment, for example, an example of the fourth generation unit is realized by using the red color determination unit 305, the low frequency region determination unit 306, and the high frequency luminance signal synthesis processing unit 308. Further, for example, by using the coefficient (α, (1.0−α)) of the equation (16), an example of a combination ratio when generating the fourth luminance signal is realized. Further, for example, an example of the fourth luminance signal is realized by using a high-frequency luminance signal.

Returning to the description of FIG. 3, the high frequency signal generation unit 309 extracts a high frequency signal by applying a band pass filter to the high frequency luminance signal output from the high frequency luminance signal synthesis processing unit 308, The high frequency signal adding unit 310 outputs the result.
The high frequency signal adding unit 310 adds the base luminance signal generated by the luminance signal synthesis processing unit 307 and the high frequency signal generated by the high frequency signal generating unit 309, and outputs a final luminance signal. Here, the high frequency signal adding unit 310 may derive a final luminance signal using the base luminance signal, the high frequency signal, and the OG signal.

  As described above, in the present embodiment, the luminance signal generation circuit 210 determines the combination ratio of the OG signal and the SWY signal according to the chromatic color determination signal Dc, and combines the OG signal and the SWY signal according to the determined combination ratio. To generate a base luminance signal. In addition, the luminance signal generation circuit 210 determines a combination ratio of the OG signal and the SWY signal according to the red determination signal Dr and the low frequency determination signal Dl, and combines the OG signal and the SWY signal according to the determined combination ratio. A luminance signal for the area is generated. Therefore, only the advantages of the OG signal and the SWY signal can be used for the base luminance signal and the high-frequency luminance signal, respectively, and a luminance signal having a wide resolvable spatial frequency region can be generated. Is possible. Therefore, the quality of the region including the high frequency region of the luminance signal can be improved.

  In this embodiment, the case where an image processing device (luminance signal generation device) is mounted on the imaging device has been described as an example. The imaging device is, for example, a digital still camera, a digital movie camera, an industrial camera, an in-vehicle camera, or a medical camera. However, as long as the apparatus has a function of performing image processing on an image captured by the imaging unit, an apparatus on which an image processing apparatus (luminance signal generation apparatus) described in the following embodiments is mounted is limited to the imaging apparatus. Not. For example, the image processing apparatus described in this embodiment may be mounted on a mobile terminal (such as a mobile phone or a tablet terminal).

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  210: luminance signal generation circuit, 304: chromatic color determination unit, 305: red determination unit, 306: low frequency region determination unit, 307: luminance signal synthesis processing unit, 308: luminance signal synthesis processing unit for high frequency

Claims (12)

A luminance signal generating apparatus that generates a luminance signal based on an image signal that is an image signal picked up by an imaging unit having a color filter and an image pickup device and is composed of a plurality of color signals including a first color signal. And
First generation means for distinguishing colors of the plurality of color signals in the image signal and generating a first luminance signal using at least the first color signal;
Second generation means for generating a second luminance signal by using the color signals input to the pixels of the image sensor without distinguishing the colors of the plurality of color signals as luminance signals of the pixels;
A combination ratio of the first luminance signal and the second luminance signal is determined based on first information based on the image signal, and the determined combination ratio, the first luminance signal, and the first luminance signal are determined. Third generation means for generating a third luminance signal based on at least one of the two luminance signals;
A synthesis ratio of the first luminance signal and the second luminance signal is determined based on information that is at least partially different from the first information as second information based on the image signal, and the determination is made A fourth luminance signal that is a luminance signal having a higher frequency than the third luminance signal is generated based on a combination ratio and at least one of the first luminance signal and the second luminance signal. A luminance signal generating device. The first information includes information on a first color in the image signal,
2. The luminance signal generation according to claim 1, wherein the second information includes at least one of information on a second color in the image signal and information on a spatial frequency in the image signal. apparatus. The information about the first color includes that the color in the image signal is a chromatic color, the color in the image signal is an achromatic color, and the degree of proximity of the color in the image signal to the achromatic color. One of
The said 3rd production | generation means determines the said synthetic | combination ratio so that the said synthetic | combination ratio of a said 2nd luminance signal may become high, so that the color in the said image signal is near achromatic color. The luminance signal generation device described.   4. The luminance signal according to claim 3, wherein when the color of the image signal is an achromatic color, the third generation unit determines the combination ratio so that the first luminance signal is not combined. 5. Generator. The information about the second color is that the color in the image signal is the color of the second color signal and that the color in the image signal is different from the color of the second color signal. One of the presence and the degree of proximity of the color in the image signal to the second color,
The fourth generation unit determines the combination ratio so that the combination ratio of the first luminance signal is higher as the color in the image signal is closer to the color of the second color signal. The luminance signal generation device according to any one of claims 2 to 4.   6. The luminance signal generation apparatus according to claim 5, wherein the color different from the color of the first color signal is red or blue.   The said 4th production | generation means determines the said synthetic | combination ratio so that the said synthetic | combination ratio of a said 1st luminance signal may become high, so that the said image signal is a low frequency signal. The luminance signal generation device according to any one of claims 6 to 6. An adjustment unit that adjusts an area of the image signal processed by at least one of the third generation unit and the fourth generation unit;
The said adjustment means performs at least any one of lowering | hanging the sensitivity in the said area | region and normalizing the said area | region with the brightness, The any one of Claims 2-7 characterized by the above-mentioned. Luminance signal generator.   The luminance signal generation apparatus according to claim 2, wherein the first color signal is a green signal. First derivation means for deriving the first information;
The luminance signal generation device according to claim 1, further comprising: a second deriving unit that derives the second information. A luminance signal generation method for generating a luminance signal based on an image signal which is an image signal captured by an imaging unit having a color filter and an imaging element and is composed of a plurality of color signals including a first color signal. And
A first generation step of distinguishing colors of the plurality of color signals in the image signal and generating a first luminance signal using at least the first color signal;
A second generation step of generating a second luminance signal using the color signal input to the pixel of the image sensor without distinguishing the colors of the plurality of color signals as a luminance signal of the pixel;
A second generation step of generating a second luminance signal by processing the color signal without distinguishing colors of the plurality of color signals;
A combination ratio of the first luminance signal and the second luminance signal is determined based on first information based on the image signal, and the determined combination ratio, the first luminance signal, and the first luminance signal are determined. A third generation step of generating a third luminance signal based on at least one of the two luminance signals;
A synthesis ratio of the first luminance signal and the second luminance signal is determined based on information that is at least partially different from the first information as second information based on the image signal, and the determination is made A fourth luminance signal that is a luminance signal having a higher frequency than the third luminance signal is generated based on a combination ratio and at least one of the first luminance signal and the second luminance signal. And a generation step of generating a luminance signal.   A program that causes a computer to function as each unit of the luminance signal generation device according to any one of claims 1 to 10.

Images