JP5383458B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to a technique suitable for use in noise suppression.

  2. Description of the Related Art Conventionally, a process for positively suppressing noise on a person's skin color has been performed. For example, in a portrait mode, the skin of a person who is a subject is reproduced beautifully by actively suppressing noise with respect to the skin color. As described above, there is known an imaging apparatus that switches the presence or absence and level of noise suppression according to the color gamut of a specific subject.

  However, when noise suppression is performed on a region that is a specific color gamut, image processing may be performed even on a region that should not be the target. As a result, the background is partially unclear. Therefore, for example, in the method described in Patent Document 1, a person image area is automatically determined so that noise suppression similar to that of a person's skin is not performed on a skin color area other than a person. Further, for example, in the method described in Patent Document 2, the noise suppression weight is determined by a vector signal having a color signal as a component in the vicinity of the region of interest, so that noise suppression is not performed on an unintended color gamut. .

JP 2004-303193 A JP 2007-110338 A

  However, the method described in Patent Document 1 is limited to human images, and the color gamut for noise suppression is limited to skin color, and there is a problem that it cannot be performed in other color gamuts. In addition, the method described in Patent Document 2 refers to neighboring colors, and thus is highly effective in the flat part of the video. However, since both color vectors are mixed in the boundary part, noise suppression is applied to an unintended color gamut. There is a problem that it cannot be avoided sufficiently.

  In view of the above-described problems, an object of the present invention is to enable appropriate noise suppression according to the color gamut distribution.

  The image processing apparatus according to the present invention includes a detection unit that detects an area occupied by the color gamut from the image data for each color gamut, and an area occupied by the first color gamut among the areas detected by the detection unit. When the area occupied by the first color gamut is equal to or greater than a first threshold, noise suppression is performed on the area occupied by the first color gamut and the area occupied by the second color gamut is: Noise suppression means for performing noise suppression on the area occupied by the second color gamut when the area occupied by the second color gamut is less than a second threshold value.

  According to the present invention, it is possible to perform appropriate noise suppression according to the color gamut distribution. Accordingly, it is possible to prevent a color region that should not be a target of noise suppression processing from becoming unclear due to noise suppression processing.

It is a block diagram which shows the structural example of the digital video camera which concerns on embodiment. It is a flowchart which shows an example of the setting procedure of a noise reduction. It is a block diagram which shows the structural example of the digital video camera which concerns on embodiment. It is a flowchart which shows an example of the process sequence of a noise reduction.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of a digital video camera 100 according to the present embodiment.
In FIG. 1, light that passes through a photographing lens 101 is incident on a CCD sensor 102 that is an image pickup device, and is incident on the CCD sensor 102 and is photoelectrically converted into an electrical signal corresponding to the amount of light. A CMOS sensor may be used instead of the CCD sensor 102. An AFE (analog front end circuit) 103 converts an analog signal output from the CCD sensor 102 into digital data and outputs the digital data to the image generation circuit 104. The image generation circuit 104 performs image processing necessary as an image such as interpolation processing and white balance, for example. Then, digital data (video data) is output to the color gamut detection circuit 105.

  The color gamut detection circuit 105 divides an image (frame) related to input video data into mesh-shaped frames, and calculates an integrated value of hue and saturation of pixels existing in each frame. Then, the microcomputer 110 sets noise reduction (NR) using the integrated values of hue and saturation calculated by the color gamut detection circuit 105. A method for setting noise reduction (noise suppression) will be described later.

  The first color gamut limited NR channel 106 performs noise reduction on the video data output from the color gamut detection circuit 105 only in a frame area determined by the microcomputer 110 as one color gamut. Further, similarly, the second color gamut limited NR channel 107 performs noise reduction limited to a frame area determined by the microcomputer 110 as one color gamut. Video data is output to the image output circuit 108 via two blocks, the first color gamut limited NR channel 106 and the second color gamut limited NR channel 107. The image output circuit 108 converts the data into a data format suitable for the recording medium and records it on the recording medium 109 as processing for output.

  Next, how noise reduction is performed by analyzing video data obtained by photographing a subject in the present embodiment will be described with reference to the flowchart of FIG. FIG. 2A is a flowchart illustrating an example of a noise reduction setting procedure in the present embodiment. Each process shown in FIG. 2A is performed under the control of the microcomputer 110.

  First, in step S <b> 201, information on the integrated value of hue and saturation for each mesh frame of video data is acquired from the color gamut detection circuit 105. Next, in step S202, the color gamut of the person's skin is set as the designated color gamut, and this designated color gamut is the first NR candidate region in accordance with the processing procedure of FIG. Determine whether the condition is met.

FIG. 2B is a flowchart illustrating an example of a processing procedure for determining the first NR candidate region. Each process shown in FIG. 2B is performed under the control of the microcomputer 110.
First, in step S221 in FIG. 2B, the number of frames included in a range preset by the hue and saturation integral values obtained in step S201 in FIG. 2A as a designated color gamut is counted. When the designated color gamut is the color gamut of the person's skin, the number of frames determined to belong to the hue area and the saturation area that are predetermined as the color gamut of the person's skin is counted.

Next, in step S222, it is determined whether or not the count number in step S221 is greater than or equal to the first threshold value. When the designated color gamut is the color gamut of the person's skin, it is determined whether or not the count number of the corresponding frame is equal to or greater than the threshold S _skin for the color gamut of the person's skin as the first threshold. As a result of this determination, if the count number is equal to or greater than the first threshold value, the process proceeds to step S223, the designated color gamut is set as the first NR candidate area, and the process ends. On the other hand, if the result of determination in step S222 is that the number of counts is less than the first threshold value, the designated color gamut is not set as the first NR candidate area, and the process ends. When the determination process of the first NR candidate region in step S202 is completed as described above, the process proceeds to the next step S203.

Returning to the description of FIG. 2A, in the next step S203, an empty color gamut is set as the designated color gamut, and the first color gamut is set for the designated color gamut in accordance with the processing procedure of FIG. It is determined whether or not a condition that is an NR candidate region is satisfied. In this case, the number of frames that are included in a range in which the integrated value of the hue and saturation obtained in step S201 is preset as an empty color gamut is counted. If the corresponding count number is equal to or greater than the _sky color gamut threshold S _sky as the first threshold, the sky color gamut is set as the first NR candidate region.

Next, in step S204, the color gamut of the evening scene is set as a designated color gamut, and whether or not the designated color gamut satisfies the condition as the first NR candidate region in accordance with the processing procedure of FIG. Determine whether or not. In this case, the number of frames included in the range set in advance as the color gamut of the evening scene is counted as the integrated value of the hue and saturation obtained in step S201. If the corresponding count number is equal to or greater than the threshold S _ss for the evening gamut as the first threshold, the evening gamut is set as the first NR candidate region.

  As described above, in the present embodiment, the color gamut of the person's skin, sky, and sunset is used as the first NR candidate region. A person's skin, sky, and evening scene are relatively smooth main subjects, and it is more likely that images with higher frequency components removed are preferred. Therefore, the skin, sky, and sunset color gamuts can be positively selected as noise reduction candidates. However, it may be the same color gamut as a person's skin, sky, and sunset scene, and may be simply a pattern of another main subject. Even if it is a person's skin, sky, and evening scene, if the proportion of the image in the image is small, the appearance of the image is not improved so much even if noise reduction is actively performed. Therefore, in this embodiment, in order to consider the degree of contribution to the appearance of the image and reduce the risk, the ratio of the human skin, sky, and sunset color gamut in the image is a predetermined value or more. Only in this case, the first NR candidate region is used.

  Next, in step S205, it is determined whether or not the number of color gamuts that are the first NR candidate areas determined in steps S202 to S204 is less than two. Here, when the color gamut of the NR candidate area is less than two, at least one of the two NR channels of the first color gamut limited NR channel 106 and the second color gamut limited NR channel 107 is used. Will be free. Therefore, it is possible to execute noise suppression processing for another color gamut using the free channel. Therefore, if the result of determination in step S205 is that there are less than two color gamuts as first NR candidate regions, the process proceeds to step S206, and if there are two or more color gamuts, the process proceeds to step S209.

  In step S209, two of the counts obtained in steps S202 to S204 are selected in order from the color gamut having the largest value, and the process proceeds to step S210.

  On the other hand, in step S206, the blue color gamut is set as the designated color gamut, and the condition that is the second NR setting candidate is set for this designated color gamut according to the processing procedure of FIG. It is determined whether or not it is satisfied.

FIG. 2C is a flowchart illustrating an example of a processing procedure for determining the second NR candidate region. Each process shown in FIG. 2C is performed under the control of the microcomputer 110.
First, in step S241, similarly to step S221 in FIG. 2B, a frame in which the integrated values of hue and saturation obtained in step S201 in FIG. 2A are included in a range set in advance as a designated color gamut. Count the number of When the designated color gamut is a blue color gamut, the number of frames determined to belong to a hue area and a saturation area that are predetermined as the blue color gamut are counted.

Next, in step S242, it is determined whether or not the count number in step S241 is less than the second threshold value. When the designated color gamut is the blue color gamut, it is determined whether or not the count number of the corresponding frame is less than the threshold S ₁ for the blue color gamut as the second threshold. As a result of this determination, if the count number is less than the second threshold value, the process proceeds to step S243, the designated color gamut is set as the second NR candidate area, and the process ends. On the other hand, if the result of determination in step S242 is that the count is greater than or equal to the second threshold value, the designated color gamut is not set as the second NR candidate region, and the process ends.

  Here, the setting of the blue color gamut as the designated color gamut is based on the characteristics of a general CCD sensor or CMOS sensor. A general CCD sensor or CMOS sensor has a low blue sensitivity with respect to other colors, and the gain tends to increase when white balance processing is performed. Since the noise increases as the gain increases, blue noise often occurs in the video. However, if the NR is set so as to suppress blue noise, there arises a problem that afterimages are generated only for the blue color or details are lost. For this reason, when the area occupied by the blue subject is large, the entire image is blurred, which makes the image look undesirable. Therefore, in the present embodiment, when there is no blue subject in the image or there are few blue subjects, the NR candidate region is selected when it is other than the main subject and the count number of the corresponding frame is less than the second threshold. I am trying to set it.

  Returning to the description of FIG. 2A, next, in step S207, it is determined whether or not the number of NR candidate regions including the first and second NR candidate regions is less than two. If the result of this determination is that there are less than two NR candidate regions, the process proceeds to step S208, and another color gamut is set as the NR candidate region. On the other hand, if the result of determination in step S207 is that there are two NR candidate areas, no more color gamuts can be set as NR candidate areas, and the process proceeds to step S210.

In step S208, the yellow color gamut is set as the designated color gamut, and whether or not the designated color gamut satisfies the condition as the second NR candidate region in accordance with the processing procedure of FIG. Determine. Here, the reason why the yellow color gamut is set as the designated color gamut is that noise generated in the image often increases for the same reason as the blue color gamut. In this case, the number of frames determined to belong to a hue area and a saturation area that are predetermined as a yellow color gamut is counted. Then, when the corresponding count number is less than the yellow color gamut threshold S ₂ as the second threshold, the yellow color gamut is set as the second NR candidate region. Then, the process proceeds to step S210.

  In step S210, two or less color gamuts satisfying one of the conditions of the first and second NR candidate regions are set as NR setting color gamuts, and the process is terminated. The number of NR setting color gamuts is two or less, and when there are two NR setting color gamuts, information on one color gamut is output to the first color gamut limited NR channel 106 and the other color gamut is output. Information is output to the second color gamut limited NR channel 107. Then, each of the first color gamut limited NR channel 106 and the second color gamut limited NR channel 107 performs noise reduction processing on an image in a frame that satisfies the color gamut.

  The first color gamut limited NR channel 106 and the second color gamut limited NR channel 107 perform the following countermeasures when the NR setting color gamut is changed by switching frames. First, the level of NR processing for the original color gamut region is gradually lowered, and after the level becomes almost zero, NR processing is started for a new color gamut region. At this time, in order to prevent a sudden change in the NR level, even when NR processing is started for a new color gamut region, time constant processing is performed so as to gradually increase the NR processing level. .

  By repeating the above processing for each frame, it is possible to perform suitable NR processing on the video. In addition, for a color gamut that is often present in a subject for which high-frequency components are desired to be removed, such as human skin, sky, and sunset scene, if the area of the color gamut region exceeds the first threshold, the target of NR processing And On the other hand, a color gamut in which noise is likely to increase according to the characteristics of the image sensor is subject to NR processing when the area of the color gamut region is less than the second threshold. Accordingly, it is possible to perform appropriate NR processing according to the subject using the color gamut as a criterion.

  In the present embodiment, the color gamut of the person's skin, the sky, and the sunset is taken as an example of the color gamut that can be the first NR candidate region, but is not limited thereto. Other color gamuts may be the target of the first NR candidate area as long as the color gamut is more desirable for the subject from which high frequency components are desired to be removed. That is, a color gamut having a higher ratio in the image data may be given priority as the first NR candidate region. In addition, the color gamut that can be the first NR candidate region may be changed according to a preset shooting mode. For example, when the portrait mode or the automatic face detection mode is set as the shooting mode, it is determined whether or not it can be the first NR candidate region only for the color gamut of the person's skin. Also good. On the other hand, when the evening scene mode is set, it may be determined whether or not it can be the first NR candidate area only for the evening gamut color gamut.

  Furthermore, in this embodiment, in order to simplify the circuit configuration, the configuration has only two color gamut limited NR channels, and is within the upper limit range according to the number of color gamuts that satisfy the condition of the first NR candidate region. Thus, the number of color gamuts that can be set as the second NR candidate region is changed. On the other hand, the number of first NR candidate regions and the number of second NR candidate regions may be fixed one by one. If the noise level of a specific color is extremely high due to the characteristics of the image sensor, the color gamut of the first NR candidate region is set after the color gamut of the second NR candidate region is set first. You may make it set. For example, when the set ISO sensitivity is high, a specification in which the color gamut of the second NR candidate region is set first is also conceivable.

  In this embodiment, the first color gamut limited NR channel 106 and the second color gamut limited NR channel 107 are used. However, the number of color gamut limited NR channels may be one, or three or more. You may have. Furthermore, in the present embodiment, a digital video camera has been described as an example, but a digital still camera can have the same configuration. It is also possible to read moving image data recorded on a hard disk or the like of a personal computer and execute the above noise reduction processing on this moving image data.

(Second Embodiment)
In the first embodiment, NR processing is performed only for the color gamut determined to be an NR candidate region. On the other hand, in the present embodiment, NR processing is originally performed for a plurality of color gamuts, and among those color gamuts that satisfy a predetermined condition, the NR level is increased. Hereinafter, a second embodiment of the present invention will be described with reference to the drawings.

FIG. 3 is a block diagram illustrating a configuration example of the digital video camera 300 according to the present embodiment. Note that the photographing lens 101, the CCD sensor 102, the AFE 103, the image generation circuit 104, the color gamut detection circuit 105, the image output circuit 108, and the recording medium 109 have the same configurations as those in FIG. .
In FIG. 3, information on the integrated values of hue and saturation calculated by the color gamut detection circuit 105 is sent to the microcomputer 310, and the video data output from the color gamut detection circuit 105 is input to the color gamut limitation NR 16 channel 306. . The color gamut limited NR 16 channel 306 performs noise reduction according to the color gamut set by the microcomputer 310 and the setting of the degree of noise suppression adapted to the color gamut.

Next, how to perform noise reduction by analyzing video data obtained by photographing a subject in the present embodiment will be described with reference to the flowchart of FIG. FIG. 4A is a flowchart illustrating an example of a noise reduction processing procedure according to the present embodiment.
First, in step S <b> 401, the microcomputer 310 acquires information on the integrated value of hue and saturation for each mesh frame of video data from the color gamut detection circuit 105. In step S402, the microcomputer 310 determines whether or not to set the 16 color gamuts to the NR enhanced color gamut according to the processing procedure of FIG. 4B described below. To do.

FIG. 4B is a flowchart illustrating an example of a processing procedure for determining the NR enhancement color gamut in step S402. Each process shown in FIG. 4B is performed under the control of the microcomputer 310.
First, in step S404 of FIG. 4B, it is determined whether or not the integrated values of hue and saturation obtained in step S401 are included in any of the 16 gamuts set in advance. The number of frames determined to be included is counted for each color gamut. In step S405, one gamut is selected from 16 preset gamuts.

  Next, in step S406, color gamut attribute data stored as a database in a ROM (not shown) is read. Then, for the color gamut selected in step S405, the attribute that increases the NR level when the frame count is greater than or equal to the threshold is the color gamut of 1, or the NR level is set when the frame count is less than the threshold. It is determined whether the strengthening attribute is a color gamut of 0. As a result of the determination, if the attribute is a color gamut of 1, the process proceeds to step S407, and if the attribute is a color gamut of 0, the process proceeds to step S409.

  Here, as an example of a color gamut having an attribute of 1, as in the first embodiment, there are many color gamuts in a subject where it is desired to remove high-frequency components such as human skin, sky, and sunset scene, It may be different depending on the shooting mode. On the other hand, an example of a color gamut having an attribute of 0 is a color gamut other than the main subject. On the other hand, as in the first embodiment, a blue or yellow color gamut that generates a relatively large amount of noise may be a color gamut having an attribute of 0, or may be different depending on the characteristics of the digital video camera 300.

  In step S407, for the color gamut having the attribute 1, it is determined whether the count number of the corresponding frame is equal to or greater than the first threshold value. As a result of this determination, if it is greater than or equal to the first threshold, the process proceeds to step S408, and if it is less than the first threshold, the process proceeds to step S411. In step S408, the color gamut in which the count number of the corresponding frame is equal to or greater than the first threshold is designated as the NR enhanced color gamut, and the process proceeds to step S411.

  On the other hand, in step S409, it is determined whether the count number of the corresponding frame is less than the second threshold for the color gamut having the attribute of 0. As a result of this determination, if it is less than the second threshold value, the process proceeds to step S410, and if it is greater than or equal to the second threshold value, the process proceeds to step S411. In step S410, the color gamut whose count is less than the second threshold is designated as the NR enhanced color gamut, and the process proceeds to step S411.

  Next, in step S411, it is determined whether or not attributes have been determined for all 16 color gamuts. As a result of the determination, when the attribute determination is performed for all 16 color gamuts, the process is ended as it is, and the process proceeds to the next step S403 in FIG. On the other hand, if the result of determination in step S411 is that there is still a color gamut for which no attribute has been determined, the process returns to step S405, and one color gamut that has not yet been selected is selected.

  Returning to the description of FIG. 4A, in the next step S403, the color gamut limited NR16 channel 306 strongly sets the NR level of the color gamut determined to be the NR enhanced color gamut, and the image in the frame is displayed. On the other hand, noise reduction processing corresponding to each color gamut is executed. The color gamut limited NR 16 channel 306 has an abrupt change in the NR level by gradually changing the level of the NR process when the NR level for the same color gamut is changed by switching frames. Time constant processing is performed so that there is no such thing.

  By repeating the above process for each frame, the noise reduction process is enhanced when the area occupied by the image is large in one color gamut, and the noise reduction process is enhanced when the area occupied by the image is small in another color gamut. Can do.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

105 color gamut detection circuit, 106 first color gamut limited NR channel, 107 second color gamut limited NR channel, 110 microcomputer

Claims (10)

Detecting means for detecting an area occupied by the color gamut from the image data for each color gamut;
Of the areas detected by the detecting means, the area occupied by the first color gamut occupies the first color gamut when the area occupied by the first color gamut is equal to or greater than a first threshold. An area occupied by the second color gamut in the area occupied by the second color gamut when the area occupied by the second color gamut is less than a second threshold while performing noise suppression on the area An image processing apparatus comprising noise suppression means for performing noise suppression on the image processing apparatus. Detecting means for detecting an area occupied by the color gamut from the image data for each color gamut;
Noise suppression means for performing noise suppression for each region detected by the detection means,
The noise suppression unit is configured such that, in a region occupied by the first color gamut among the regions detected by the detection unit, the area of the region occupied by the first color gamut is equal to or greater than a first threshold. Further, noise suppression is performed more strongly on the area occupied by the first color gamut than when the area is less than the first threshold, and in the area occupied by the second color gamut, the second color gamut is When the area occupied is less than the second threshold, noise suppression is performed more strongly on the area occupied by the second color gamut than when the area is greater than or equal to the second threshold. Image processing device.   3. The image processing apparatus according to claim 1, wherein the noise suppression unit preferentially sets a color gamut having a larger ratio to the image data as the first color gamut. 4.   The image processing apparatus according to claim 1, wherein the first color gamut is a color gamut that varies depending on a shooting mode.   The image processing apparatus according to claim 1, wherein the noise suppression unit prioritizes noise suppression for an area occupied by the first color gamut when there is an upper limit in the number of color gamuts for which noise suppression is performed. .   6. The device according to claim 1, wherein the first color gamut is a color gamut related to a main subject, and the second color gamut is a color gamut related to a region other than the main subject. Image processing device. A detection step for detecting an area occupied by the color gamut from the image data for each color gamut;
Of the areas detected in the detection step, the area occupied by the first color gamut occupies the first color gamut when the area occupied by the first color gamut is equal to or greater than a first threshold. An area occupied by the second color gamut in the area occupied by the second color gamut when the area occupied by the second color gamut is less than a second threshold while performing noise suppression on the area And a noise suppression step of performing noise suppression on the image processing method. A detection step for detecting an area occupied by the color gamut from the image data for each color gamut;
A noise suppression step of performing noise suppression for each region detected in the detection step,
In the noise suppression step, in the region occupied by the first color gamut among the regions detected in the detection step, the area of the region occupied by the first color gamut is equal to or greater than a first threshold. Performs noise suppression more strongly on the area occupied by the first color gamut than in the case where the area is less than the first threshold, and in the area occupied by the second color gamut, the second color gamut When the area occupied by the area is less than a second threshold, noise suppression is performed more strongly on the area occupied by the second color gamut than when the area is greater than or equal to the second threshold. Image processing method. A detection step for detecting an area occupied by the color gamut from the image data for each color gamut;
Of the areas detected in the detection step, the area occupied by the first color gamut occupies the first color gamut when the area occupied by the first color gamut is equal to or greater than a first threshold. An area occupied by the second color gamut in the area occupied by the second color gamut when the area occupied by the second color gamut is less than a second threshold while performing noise suppression on the area A program for causing a computer to execute a noise suppression step for performing noise suppression on a computer. A detection step for detecting an area occupied by the color gamut from the image data for each color gamut;
Causing the computer to perform a noise suppression step for performing noise suppression for each region detected in the detection step,
In the noise suppression step, in the region occupied by the first color gamut among the regions detected in the detection step, the area of the region occupied by the first color gamut is equal to or greater than a first threshold. Performs noise suppression more strongly on the area occupied by the first color gamut than in the case where the area is less than the first threshold, and in the area occupied by the second color gamut, the second color gamut When the area occupied by the area is less than a second threshold, noise suppression is performed more strongly on the area occupied by the second color gamut than when the area is greater than or equal to the second threshold. Program to do.

Images