JP5501393B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  2. Description of the Related Art Conventionally, in an image processing apparatus that performs compression coding on a captured image, a process including detecting a region including a predetermined color that is considered to be highly important and improving the image quality of the region where the predetermined color is detected There is a device to do.

  In many cases, a color that is considered to be highly important in such an apparatus is generally a color corresponding to a human face, that is, a skin color.

  As a proposal for improving image quality by detecting a region including skin color in an image in this way, performing filter processing on the region where skin color is detected, and controlling a code allocation amount at the time of image compression There is one.

  FIG. 15 is a block diagram of an image processing apparatus according to a conventional example (Patent Document 1).

  In FIG. 15, the imaging unit 1501 includes a lens and an imaging element and captures (captures) an image. The image signal processing unit 1502 executes a series of image signal processing such as gamma conversion, color space conversion, and raster-block conversion.

  The skin color detection processing unit 1503 detects, in particular, the face portion of a person who is generally highly important as a subject to be photographed with respect to an image that has undergone a series of image processing. The filter unit 1504 selectively attenuates the high-frequency component intensity for an arbitrary region of the captured image data.

  The motion compensation prediction unit 1505 predicts a reference image block that most closely matches the prediction target image block, and generates a motion vector. The orthogonal transform unit 1506 performs orthogonal transform such as discrete cosine transform. The quantization unit 1507 divides the orthogonal transform coefficient by a quantization matrix or the like to reduce the information amount.

  The entropy encoding unit 1508 further reduces the amount of information by performing variable length encoding using a Huffman code or the like. The buffer unit 1509 smoothes the amount of code generated per unit time that fluctuates on the time axis.

  The recording unit 1510 records the encoded data on a recording medium such as a magnetic tape or a memory card. The inverse quantization unit 1511 performs inverse quantization on the orthogonal transform coefficient quantized by the quantization unit 1507 in order to reconstruct the reference image.

  The inverse orthogonal transform unit 1512 performs inverse orthogonal transform on the orthogonal transform coefficient after inverse quantization generated by the inverse quantization unit 1511. The shooting mode selection unit 1513 is operated by the photographer to select (set) a shooting mode.

  The control unit 1514 monitors the occupation ratio of the buffer unit 1509 and controls the entire system in an integrated manner. The encoding mode selection unit 1515 is operated by the photographer and selects a still image or a moving image.

  FIG. 16 is a flowchart illustrating a procedure of code amount control processing in the quantization unit of FIG.

  When the photographer gives an instruction to start shooting in step S1601, the control unit 1514 checks the encoding mode set in the encoding mode selection unit 1515 in step S1602.

  If the encoding mode is the moving image encoding mode, the process advances to step S1603 to check the shooting mode set in the shooting mode selection unit 1513. If the encoding mode is not the moving image encoding mode, the process ends.

  In step S1603, if the shooting mode is not the portrait mode, the process ends.

  On the other hand, if the shooting mode is the portrait mode in step S1603, the process proceeds to step S1604. In step S1604, the code amount is increased by decreasing the quantization step value when the image block corresponding to the area detected as the skin color in the skin color detection processing unit 1503 is quantized, and the process ends.

  In addition to this control, when quantizing an image block corresponding to a region other than the region detected as skin color, the quantization unit 1507 is set so as to reduce the code amount by increasing the quantization step value or the like. Control. As a result, the quantization unit 1507 is controlled so that the code amount for the region detected as the skin color in the skin color detection processing unit 1503 is relatively increased as compared with the other regions.

  By such code amount control processing, the code amount in the region where the presence of flesh color is detected in the image is increased, while the code amount in other regions is reduced, so skin color is included while maintaining the entire code amount. The configuration is such that the image quality of the area can be improved.

  Japanese Patent Laid-Open No. 2004-228620 has a configuration in which the code amount can be effectively reduced by giving the code amount to the area where the presence of skin color is detected only when the user selects the portrait mode.

Japanese Patent Laid-Open No. 2005-218017

  The problems in the prior art disclosed in Patent Document 1 will be described below.

  Normally, in an image processing apparatus represented by a video camera as shown in FIG. 15, so-called white balance correction is performed for the purpose of correcting a change in color temperature of a light source that illuminates a subject.

  Here, in order to explain the problem to be solved by the present invention, a general white balance correction operation will be described.

(General white balance operation explanation)
The color temperature range that can be controlled by white balance correction is set assuming a color temperature change range of a light source used in actual photographing, and is limited to a certain finite range.

  Usually, as a color temperature range in which white balance can be corrected, the high color temperature side is set near the color temperature of sunlight, and the low color temperature side is set near the color temperature of the incandescent light bulb.

  In this way, by setting a color temperature range that can be controlled by white balance correction, for example, when shooting a subject illuminated by a light source with a low color temperature, such as a candle or a flame of a fireplace, the redness of the light source Becomes a left image, and a warm image can be obtained.

  On the other hand, when a subject is photographed under a light source with a high color temperature, such as in the shade, an image in which the light source is left bluish is left, and a refreshing image can be obtained.

  Furthermore, by setting the color temperature range that can be controlled by white balance correction within a predetermined finite range, the following effects can be obtained. In other words, even when white balance miscorrection called color failure occurs due to the influence of a specific chromatic color, there is an effect of preventing a situation in which the color of an image subjected to white balance correction is extremely shifted.

  In addition, in image processing apparatuses represented by video cameras that shoot continuous video in time, the amount of correction per unit time is limited so that the color does not change during the scene due to abrupt white balance correction. The correction is gentle over time.

  A problem in the case of detecting a region having the above-described white balance correction function and including skin color will be described.

  FIG. 17 is a diagram showing a subject, and is an example of a person wearing white clothes. In the figure, reference numerals 1701 and 1702 denote a human face portion and a white clothing portion, which are skin color portions, respectively.

  FIG. 18 is a vector diagram showing the level of the color difference signal in the image signal when the subject shown in FIG. 17 is photographed.

  In the state in which the white balance is corrected, the clothing portion 1702 of the subject is white, so the point is indicated by P1802 in FIG.

  Since the face portion 1701 of the subject has a flesh color, the point is indicated by P1801 in FIG. In FIG. 18, a broken line A1801 indicates a threshold value of the color difference signal for determining the skin color. If a color difference signal exists in the inner area surrounded by the broken line A1801, it is determined that the skin color. Hereinafter, this region is referred to as a skin color determination region A1801.

  When the color difference signal of the image is in the state shown in FIG. 18, P1801, which is the human face 1701, is determined to be skin color because it is inside the skin color determination area A1801.

  Further, P1802 which is the clothing portion 1702 is not determined to be a skin color because it is not inside the skin color determination area A1801.

  In the conventional method in which a large amount of code is assigned to an area including a skin color, the face 1701 in FIG. 17 is determined to be a skin color in a state in which such normal white balance correction is performed. A relatively large amount of code is assigned.

  In addition, since the clothing portion 1702 in FIG. 17 is not determined to be a skin color, a relatively small code amount is allocated, and the image quality of the region including the skin color can be improved while maintaining the entire code amount.

  However, when the color temperature is lower or higher than the normal white balance control range, the following problem occurs.

  FIG. 19 is a vector diagram showing the level of the color difference signal when the person shown in FIG. 17 is photographed under a low color temperature light source.

  For example, when lit by a light source with a color temperature lower than the normal white balance control range, such as a candle or a fireplace flame, that is, when the white balance is operating at a low color temperature side, the white balance Even if correction is made, the redness of the image may remain.

  In this case, the clothing portion 1702 of the subject is a point indicated by P1902 in FIG. Further, the face portion 1701 of the subject is a point indicated by P1901 in FIG. Note that A1901 in FIG. 19 is the same skin color determination area as A1801 in FIG.

  In the state shown in FIG. 19, since P1901 which is the face part of a person and the clothes part P1902 are inside the skin color determination area A1901, the skin color is determined. As a result, an operation is attempted to assign a large amount of code to both the face portion 1701 and the clothing portion 1702.

  However, if a large amount of code is assigned to the garment portion 1702 in addition to the face portion 1701, a large amount of code amount cannot be assigned only to the face portion 1701, thereby improving the image quality of the face portion 1701. There is a problem that it becomes impossible.

  In addition, when the light source has a low color temperature, even if the subject is not the skin of the original person, other subjects tend to have a reddish color similar to the skin color. In this case, when the filter process is performed on the skin color determination region, the filter process is performed on other subjects similar to the skin color, and the image quality other than the person's skin color part is also changed.

  FIG. 20 is a vector diagram showing the color difference level of the image signal when the person shown in FIG. 17 is photographed under a high color temperature light source.

  For example, when the image is illuminated with a light source having a high color temperature such as a shade, that is, when the white balance is operated on the high color temperature side, the blueness of the image may remain.

  In this case, the clothing portion 1702 of the subject is a point indicated by P2002 in FIG. Further, the face portion 1701 of the subject is a point indicated by P2001 in FIG. In FIG. 20, A2001 is the same skin color determination area as A1801 in FIG.

  In this case, neither P2001, which is a person's face portion, nor clothing portion P2002 is within the skin color determination area A2001, and therefore is not determined to be a skin color. As a result, there is a problem that the image quality of the face portion 1701 cannot be improved because an operation for assigning a large amount of code to the face portion 1701 is not performed.

  This problem may occur even when the color temperature is extremely low or high as described above.

  As described in the explanation of the white balance correction operation, the white balance correction is controlled so as to change gradually with time. Therefore, when the color temperature of the light source changes abruptly, for example, when the camera moves from indoor to outdoor, white balance correction may not be able to follow.

  For example, assume that the subject in FIG. 17 is photographed indoors with a low color temperature, and the subject has moved to the outdoors with a high color temperature in a state where the white balance is correctly corrected as shown in FIG.

  In that case, since the color temperature of the light source changes abruptly high, the color difference vector temporarily enters the state shown in FIG.

  After that, until the white balance follows the outdoor color temperature and is correctly corrected as shown in FIG. 18, the skin color is shifted to the blue side from the original, so that the face of the person is not correctly determined as the skin color.

  Similarly, when moving from the outdoor where the color temperature is high to the indoor where the color temperature is low, the color difference vectors are temporarily in a state as shown in FIG.

  Thereafter, until the white balance follows the outdoor color temperature and is corrected correctly as shown in FIG. 18, clothes other than the skin color are also erroneously determined as the skin color.

  As described above, even when the color temperature of the light source changes abruptly, in the conventional example, the face portion is not correctly detected, and the operation of assigning a large amount of code only to the face portion, that is, the skin color portion is not performed. There was a problem.

The object of the present invention is to detect the face correctly even when the color temperature of the light source changes abruptly when the white balance is operating in the auto mode, and to stabilize the image quality of the area including the skin color. An object of the present invention is to provide an image processing apparatus that can be improved.

In order to achieve the above object, an image processing apparatus according to claim 1 includes a color detection unit that detects an area including a predetermined color in an image, a filter processing unit that performs a filtering process on image data, and image data. A white balance adjusting means for adjusting a white balance control state over a predetermined time when the color temperature of the light source at the time of photographing changes, and a white balance control adjusted by the white balance adjusting means A determination unit that determines whether or not the state is a low color temperature side, and a luminance signal and a threshold value of the color signal when the predetermined color is detected in the color detection unit are changed according to a determination result by the determination unit. and a control unit, wherein the control means, when the white balance control state determines that the determination means that the low color temperature side, the white balance control form There compared with the case where the determination means is not the low color temperature side is determined, in the photographing after the captured image, a threshold saturation is changed to a high direction in detecting skin color in the color detection unit, and, Depending on the difference between the currently adjusted white balance control state and the finally adjusted white balance control state when the light source during shooting changes during continuous shooting performed in a time shorter than the predetermined time Te, characterized that you determine the degree of change of the threshold.

The image processing apparatus according to claim 5, a color detection unit that detects an area including a predetermined color from input image data, a filter processing unit that performs a filtering process on the image data, and a color temperature of a light source at the time of photographing and white balance adjustment means, wherein said white balance control state adjusted by the white balance adjusting means is in the low color temperature side for adjusting the white balance control state obtained from the captured image over a predetermined time period when but has changed Determination means for determining whether or not, and a control means for changing a threshold value of a luminance signal and a color signal when detecting the predetermined color in the color detection unit according to a determination result by the determination means , wherein, when the white balance control state determines that the determination means that the low color temperature side, the white balance control state low color temperature side Compared to the case where the determination means determines that there is no, in the photographing after the captured image, a threshold saturation was the change in the high direction in detecting skin color in the color detection unit, and shorter than the predetermined time When the light source at the time of shooting changes during continuous shooting performed in time, the threshold value is set according to the difference between the currently adjusted white balance control state and the finally adjusted white balance control state. to determine the degree of change, characterized in Rukoto.

The image processing apparatus according to claim 8, a color detection unit that detects an area including a predetermined color in the image, a filter processing unit that performs a filtering process on the image data, and an encoding unit that encodes the image data White balance adjustment means for adjusting the white balance control state obtained from the photographed image; and determination means for determining whether or not the white balance control state adjusted by the white balance adjustment means is a low color temperature side; Control means for changing a code amount at the time of image encoding assigned to an area in which the predetermined color is detected in the encoding unit according to a determination result by the determination means, and the control means includes the white balance If the control state is determined the determination means that the low color temperature side, the photographing after the captured image, is low the white balance control state Characterized in that to reduce the code amount in image coding to be allocated to the skin color is detected area in the encoding unit compared to when the determination means that no temperature side is the high color temperature side was determined.

The image processing apparatus according to claim 10, a color detection unit that detects an area including a predetermined color in an image, a filter processing unit that performs a filtering process on the image data, and an encoding unit that encodes the image data White balance adjustment means for adjusting the white balance control state obtained from the photographed image; and determination means for determining whether or not the white balance control state adjusted by the white balance adjustment means is a low color temperature side; Control means for changing the type and intensity of the filter processing applied to the area where the predetermined color is detected in the filter processing unit in accordance with the determination result by the determination means, and the control means includes the when the white balance control state determines that the determination means that the low color temperature side, the photographing after the captured image, the white bar Nsu control state is equal to or to weaken the strength of the filtering process performed on skin color is detected region in the filtering process unit compared with the case where the determination means is not the low color temperature side is determined.

According to the image processing apparatus of the present invention, when the white balance is operating in the auto mode, even when the color temperature of the light source changes abruptly , the face is correctly detected and the image quality of the region including the skin color is detected. Can be improved more stably.

FIG. 1 is a block diagram showing a first configuration example of an image processing apparatus represented by a video camera of the present invention. It is a figure which shows the example in which the auto mode (auto white balance mode) is selected in the white balance operation mode selection part in FIG. FIG. 2 is a diagram illustrating a relationship between white balance control data and color temperature in the image processing apparatus of FIG. 1. FIG. 2 is a color difference vector diagram illustrating a relationship between a color temperature change and a skin color determination region in the image processing apparatus of FIG. 1. FIG. 2 is a color difference vector diagram illustrating a relationship between a color temperature change, a skin color determination region, a white portion, and a skin color portion in the image processing apparatus of FIG. It is a flowchart which shows the procedure of the filter process and quantization process which are performed by the image processing apparatus of FIG. It is a figure which shows the relationship between the color temperature change and quantization correction coefficient in the image processing apparatus of FIG. FIG. 2 is a color difference vector diagram showing a relationship between a white balance operation mode and a skin color determination region in the image processing apparatus of FIG. 1. It is a figure which shows the present value and target control value of white balance control data in the image processing apparatus of FIG. FIG. 2 is a color difference vector diagram illustrating a relationship between a target control value and a skin color determination region in the image processing apparatus of FIG. 1. It is a block diagram which shows the 2nd structural example of the image processing apparatus represented by the video camera of this invention. It is a figure which shows the image data and metadata on the recording medium in FIG. It is a flowchart which shows the procedure of the white balance detection process in the white balance detection part in FIG. It is a flowchart which shows the procedure of the filter control process of the filter part in FIG. It is a block diagram of the image processing apparatus which concerns on a prior art example (patent document 1). 16 is a flowchart illustrating a procedure of code amount control processing in the quantization unit in FIG. 15. Is a diagram showing a subject. FIG. 18 is a vector diagram showing the level of a color difference signal in an image signal when the subject shown in FIG. 17 is captured. FIG. 18 is a vector diagram showing the level of a color difference signal when the person shown in FIG. 17 is photographed under a low color temperature light source. FIG. 18 is a vector diagram showing a color difference level of an image signal when the person shown in FIG. 17 is photographed under a high color temperature light source.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a first configuration example of an image processing apparatus represented by a video camera of the present invention.

  In FIG. 1, an imaging unit 101 includes a lens and an image sensor and captures (captures) an image. The image signal processing unit 102 executes a series of image signal processing such as gamma conversion, color space conversion, white balance processing, and raster-block conversion.

  A skin color detection processing unit (color detection unit) 103 detects, in particular, a face portion of a person who is generally highly important as a subject to be photographed with respect to an image subjected to a series of image processing. The filter unit (filter processing unit) 104 selectively attenuates the high-frequency component intensity for an arbitrary region of the captured image data.

  The motion compensation prediction unit 105 predicts a reference image block that most closely matches the prediction target image block, and generates a motion vector. The orthogonal transform unit 106 performs orthogonal transform such as discrete cosine transform. The quantization unit 107 divides the orthogonal transform coefficient by a quantization matrix or the like to reduce the information amount.

  The entropy encoding unit 108 further reduces the amount of information by performing variable length encoding using a Huffman code or the like. The buffer unit 109 smoothes the amount of code generated per unit time that fluctuates on the time axis.

  The recording unit 110 records the encoded data on a recording medium such as a magnetic tape or a memory card. The inverse quantization unit 111 performs inverse quantization on the orthogonal transform coefficient quantized by the quantization unit 107 in order to reconstruct the reference image.

  The inverse orthogonal transform unit 112 performs inverse orthogonal transform on the orthogonal transform coefficient after the inverse quantization generated by the inverse quantization unit 111. The white balance operation mode selection unit 113 selects (sets) a white balance operation mode operated by the photographer.

  The control unit (control unit) 114 controls the video camera in an integrated manner. That is, the control unit 114 performs control to monitor the occupation rate of the buffer unit 109 and maintain the encoded data amount, threshold control in skin color detection processing described later, intensity control of the filter unit 104, and the like.

  The control unit 114 changes the threshold value of the luminance signal and the color signal when detecting the predetermined color in the skin color detection processing unit 103 according to the white balance control state and the color temperature information at the time of image capturing. Or the control part 114 changes the code amount at the time of the image encoding allocated to the area | region where the predetermined color in the entropy encoding part 108 was detected. Alternatively, the control unit 114 changes the type and intensity of the filtering process performed on the area where the predetermined color is detected in the filter unit 104.

  Further, the control unit 114 detects the skin tone color and the threshold value of the color signal in the skin color detection processing unit 103 when the white balance control state is fixed to a predetermined value and when the white tracking control state is the automatic tracking state. To change.

  Further, the control unit 114 obtains the current correction amount and the final correction amount in the white balance control, and the luminance signal when the predetermined color is detected in the skin color detection processing unit 103 according to the difference between the correction amount and the final correction amount. And the threshold value of the color signal is changed.

  A characteristic operation of the present invention in the image processing apparatus having the above configuration will be described.

  The image data input from the imaging unit 101 is subjected to a series of image signal processing such as gamma conversion, color space conversion, white balance processing, and raster-block conversion in the image signal processing unit 102.

  The white balance processing here operates according to the operation mode set by the white balance operation mode selection unit 113.

  FIG. 2 is a diagram illustrating an example in which an auto mode (auto white balance mode) is selected by the white balance operation mode selection unit in FIG.

  As shown in FIG. 2, a white balance operation mode that can be set includes a so-called auto white balance mode in which the color temperature change of the light source is automatically tracked and adjusted. Also, white balance set mode for shooting white paper and adjusting to the color temperature of any light source, preset mode that can be set to white balance adjustment values that match the color temperature of several light sources, such as sunlight and fluorescent lights Etc.

  The operation mode of white balance selected in this way, the operation state of white balance in the image signal processing unit 102, and the color temperature information are output to the control unit 114.

  Next, the skin color detection processing unit 103 performs skin color detection processing on the image subjected to a series of image processing for the purpose of detecting a person who is generally high in importance as a photographing target.

  Here, the control state of the white balance and the operation of the skin color detection processing unit 103 and the control unit 114 in the present embodiment will be described with reference to FIGS. 3, 4, and 5.

  FIG. 3 is a diagram illustrating a relationship between color temperature and white balance control data in the image processing apparatus of FIG.

  The characteristics shown by the solid line in FIG. 3 are the control characteristics of the white balance control data R_Gain and B_Gain accompanying the change in color temperature, R_Gain is data for adjusting the gain of the red component in the image signal, and B_Gain is in the image signal. This is data for adjusting the gain of the blue component of.

  As shown in FIG. 3, R_Gain and B_Gain change from RG1 to RG4 and BG1 to BG4, respectively, as the color temperature changes from the low color temperature to the high color temperature.

  The control unit 114 determines that the white balance control state is the low color temperature side when R_gain is RG2 or less and / or B_gain is BG2 or more. When R_gain is RG3 or more and / or B_gain is BG3 or less, it is determined that the white balance control state is on the high color temperature side. When the white balance control state is neither the high color temperature side nor the low color temperature side, the control unit 114 determines that the normal range is reached.

  4A to 4C are diagrams illustrating the relationship between the color temperature determined by the control unit in FIG. 1 and the skin color determination region by the skin color detection processing unit.

  A region A401a within the broken line in FIG. 4A is a skin color determination region when the color temperature is determined to be the low color temperature side.

  A region A401b within the broken line in FIG. 4B is a skin color determination region when the color temperature is determined to be the normal range.

  A region A401c in the broken line in FIG. 4C is a skin color determination region when the color temperature is determined to be the high color temperature side.

  In this way, the skin color determination area A401a when determined to be on the low color temperature side is compared to the skin color determination area 401b when determined to be the normal range of FIG. 4B, as shown in FIG. Shift toward higher saturation.

  Further, the skin color determination area A401c when determined to be on the high color temperature side shifts in a direction of lower saturation as shown in FIG. 4C.

  As described above, the skin color detection processing unit 103 according to the present embodiment performs an operation of changing the skin color determination region in accordance with the white balance control state.

  Here, the skin color detection operation when the color temperature is low and high according to the present embodiment will be described with reference to FIG.

  FIG. 5A is a vector diagram showing the color difference level of the image signal when the person shown in FIG. 17 is photographed with a low color temperature light source.

  As described above, for example, when a subject is illuminated with a light source having a low color temperature such as a candle or a fireplace flame, that is, when the white balance is operating on the low color temperature side, the white balance correction is performed. Even if it is done, the redness of the image may remain.

  In this case, the clothing portion 1702 (FIG. 17) of the subject is a point indicated by P502a in FIG. Further, the face portion 1701 (FIG. 17) of the subject is a point indicated by P501a in FIG.

  However, as shown in A501a, the skin color determination area in the present embodiment shifts in the direction of higher saturation, so only the face color difference signal P501a enters the skin color determination area A501a, and the color difference signal of the clothing part. P502a is not determined to be a skin color.

  FIG. 5C is a vector diagram showing the color difference level of the image signal when the person shown in FIG. 17 is photographed with a high color temperature light source.

  As described above, for example, when a subject is illuminated with a light source having a high color temperature such as a shade, that is, when the white balance is operating on the high color temperature side, the image is displayed even if white balance correction is performed. The blueness of may remain.

  In this case, the clothing part 1702 of the subject is a point indicated by P502c in FIG.

  Further, the face portion 1701 of the subject is a point indicated by P501c in FIG. 5C, but the skin color determination area of the present embodiment shifts in a direction of lower saturation as indicated by A501c. P501c, which is the face portion, can be determined as the skin color.

  If the white balance control state is in the normal range shown in FIG. 5B, only P501b, which is a human face, can be determined to be a skin color as in the conventional example. P502b which is the clothing part 1702 is not determined to be a skin color.

  Thus, by changing the threshold value of the color difference signal for determining the skin color in accordance with various color temperature states, it becomes possible to detect the skin color stably regardless of the color temperature of the light source.

  FIG. 6 is a flowchart illustrating a procedure of filter processing and quantization processing executed by the image processing apparatus of FIG.

  This process is executed under the control of the control unit 114 in FIG.

  In FIG. 6, when processing is started, first, in step S601, a basic quantization step Q is calculated based on the target code amount of the entire image. Next, in step S602, the color temperature is determined from the white balance control state.

  Step S602 functions as a determination unit that determines the color temperature from the white balance control state. As shown in FIG. 3, the color temperature is determined based on the values of the white balance control data R_Gain and B_Gain. In step S603, it is determined whether the color temperature is on the low color temperature side.

  If it is determined in step S603 that the color temperature is lower, the quantization step Q is multiplied by the correction coefficient K1 in step S604 to calculate the corrected quantization step Q ′, and the filter strength is reduced in step S605. Set.

  If it is not determined in step S603 that the color temperature is low, it is determined in step S606 whether the color temperature is high. If it is determined in step S606 that the color temperature is higher, a correction quantization step Q ′ obtained by multiplying the quantization step Q by the correction coefficient K3 is calculated in step S607, and the filter strength is increased in step S608. Set.

  If it is not determined in step S606 that the color temperature is higher, the normal color temperature is assumed, and in step S609, a correction quantization step Q ′ obtained by multiplying the quantization step Q by the correction coefficient K2 is calculated, and in step S610. Set the filter strength to medium.

  Steps S604, S607, and S609 function as a correction unit that corrects the quantization step calculated based on the target code amount of the entire image based on the determination result of the determination unit.

  Steps S605, S608, and S610 function as a setting unit that sets the filter strength based on the correction result of the correction unit.

  Note that the values of the above-described coefficients K1, K2, and K3 have the relationship shown in Formula 1.

[Equation 1]
K3 <K2 <K1 <1.0
By giving the coefficient values of K1 to K3 shown in Equation 1, the corrected quantization step Q ′ becomes smaller than the basic quantization step Q. As the color temperature increases, the correction quantization step Q ′ is set smaller.

  Also, the filter strength is set stronger as the color temperature increases. In the present embodiment, as the filter strength increases, an operation for further reducing high-frequency components in the image signal is performed.

  Next, in step S611, filter processing is performed on an area determined to include skin color using the filter strength set in the above-described processing.

  Step S611 functions as a filter processing unit that filters the skin color determination region with the filter strength set by the setting unit.

  Next, in step S612, a quantization process is performed on an area determined to include skin color using the correction quantization step Q 'set in the above process.

  Step S612 functions as a quantization processing unit that quantizes the skin color determination region using the corrected quantization step.

  Although not shown in FIG. 6, quantization is performed using a basic quantization step Q for portions other than the skin color determination region.

  Thereafter, after a series of encoding processes are performed by the entropy encoding unit 108, the encoded data is temporarily stored in the buffer unit 109 and finally recorded on a recording medium such as a magnetic tape or a memory card by the recording unit 110. The

  As described above, according to the present embodiment, the luminance signal and the color signal threshold when detecting the skin color, the filter strength and the quantization performed on the skin color determination area according to the color temperature information of the light source and the white balance control state. The step can be changed in a timely manner.

  As a result, it is possible to accurately and stably recognize especially the face portion of a person who is generally highly important as an imaging target without being affected by the color temperature change by the skin color detection process.

  Further, the inverse orthogonal transform unit 112 can increase the amount of information by reducing the quantization step value when quantizing the image block corresponding to the face part according to the color temperature.

  In addition, since the filter processing corresponding to the color temperature is performed on the skin color determination region, it is possible to reduce the change in image quality of subjects other than a person due to the filter processing when the color temperature is low.

  As a result, it is possible to stably improve the image quality of the region including the skin color.

  In the present embodiment, the threshold for detecting skin color, the quantization step, and the filter strength are changed at three levels according to the color temperature. However, the level may not be particularly three.

  For example, as shown by a solid line in FIG. 7, by continuously changing the quantization correction coefficient according to R_Gain that is white balance control data that changes in response to a change in the color temperature of the light source, The quantization step may be continuously changed.

  It goes without saying that the color difference signal and the threshold level of the luminance signal that determine the skin color determination region according to the color temperature, and the filter strength may be continuously changed.

  In this embodiment, an example of an encoding method that can change the quantization step has been described. However, in the case of an encoding method that cannot change the quantization step, only the filter strength may be changed. .

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. Note that the configuration of the image processing apparatus according to the second embodiment is the same as that of the first embodiment, and a description thereof will be omitted.

  The feature of the present embodiment is that the skin color determination region is changed according to the white balance control state, particularly, the white balance operation mode.

  As described with reference to FIG. 2, the white balance operation mode that can be set includes a so-called auto white balance mode that automatically tracks and adjusts the color temperature change of the light source. Also, white balance set mode for shooting white paper and adjusting it to the color temperature of any light source, preset mode that can be set to white balance adjustment values that match the color temperature of several light sources, such as sunlight and fluorescent lights, etc. There is.

  In the white balance set mode, a white subject such as white paper is once photographed to adjust the white balance, so the white balance can be corrected more accurately than in the auto white balance mode. Also, when using a preset mode such as sunlight or fluorescent light, the photographer can use the light source after knowing the color temperature state of the light source in advance, so that the white balance can be corrected relatively correctly.

  On the other hand, in the auto white balance mode, it is easy to be influenced by the color of the subject, and when the person is photographed, the skin color saturation may be slightly reduced.

  The present embodiment is for dealing with the difference in skin color saturation between the auto white balance mode and the other modes.

  FIG. 8 is a diagram showing changes in the white balance operation mode and the skin color determination area in the present embodiment.

  A801a in FIG. 8A indicates a skin color determination region when the white balance operation mode is set mode or preset mode. In FIG. 8A, P801a and P802a correspond to the face portion 1701 and the clothing portion 1702 of the place where the person shown in FIG.

  As shown in FIG. 8A, when the white balance operation mode is set mode or preset mode, the photographer correctly adjusts the white balance according to the color temperature of the light source. P802a to be aligned with the center of the vector diagram.

  In this embodiment, in such a case, the skin color determination area A801a is set as a fixed area without being changed.

  On the other hand, in FIG. 8B, P801b and P802b correspond to the face portion 1701 and the clothing portion 1702 when the person shown in FIG.

  As shown in FIG. 8B, when the white balance operation mode is the auto mode, the white balance is slightly erroneously corrected, and the point P802b corresponding to white in the clothing portion 1702 of the subject is slightly in the blue direction from the center of the vector diagram. Shift. And in connection with it, the saturation of the point P801b equivalent to the face part 1701 may also fall.

  In the present embodiment, when the white balance operation mode is the auto mode, as shown in FIG. 8B, the white balance operation is performed by shifting the skin color determination area A801b slightly toward the low saturation side. Regardless of the mode, it becomes possible to correctly detect the skin color.

  As a result, as in the first embodiment, the face portion of a person who is generally highly important as an object to be photographed is recognized with high accuracy and stability without being affected by a change in color temperature by the skin color detection process. can do.

  Further, according to the color temperature, the inverse orthogonal transform unit 112 can reduce the quantization step value and increase the amount of information when quantizing the image block corresponding to the face part. Since the skin color determination region is also subjected to filter processing according to the color temperature, as a result, the code amount for the region detected as the skin color by the skin color detection processing unit 103 is relatively smaller than the other regions. The quantization unit 107 can be controlled to increase. As a result, it is possible to stably improve the image quality of the region including the skin color.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described. Note that the configuration of the image processing apparatus according to the third embodiment is the same as that of the first embodiment, and a description thereof will be omitted.

  In this embodiment, in addition to the configuration and operation of the first embodiment, when the white balance is operating in the auto mode, it is possible to appropriately cope with the case where the color temperature of the light source changes suddenly. Is.

  The feature of the present embodiment is that the skin color determination region is changed in accordance with the difference between the target correction amount of the white balance control data and the current correction value.

  The third embodiment will be described in detail with reference to FIGS. 9 and 10.

  Here, it is assumed that the color temperature of the light source is high and the white balance control data is white balance corrected by outputting the value of TS in FIG.

  In this case, the color difference vectors of the face portion 1701 and the white clothing portion 1702 when the subject of FIG. 17 is photographed are P_TS1 and P_TS2 in FIG. Further, the skin color determination area is A_TS in FIG.

  Thereafter, when the color temperature of the light source rapidly changes, the color difference vectors shift from P_TS1 in FIG. 10 to P_TD1 and P_TS2 to P_TD2, respectively. This is because the white balance correction is performed slowly in time, so that the white balance correction cannot follow when the color temperature of the light source changes rapidly.

  In this case, with only the configuration of the first embodiment, the skin color determination area remains A_TS in FIG. 10 until the white balance follows, so P_TD1 corresponding to the skin color after the color temperature change is There is a risk that it will be outside the A_TS area and will not be detected as skin color.

  Therefore, in the present embodiment, the skin color determination region is changed according to the difference between the current correction value of the white balance control data and the target correction value to be finally set.

  The target correction value for white balance is obtained by calculating how much the value of the color difference signal in the area corresponding to white, such as a white clothing part, is from the center of the color difference vector (point of zero color difference). be able to. In this embodiment, it is obtained from the amount of deviation from the center of the color difference vector of P_TD2 in FIG.

  Here, when the target correction amount of the obtained white balance control data is TD in FIG. 9, the skin color determination region is shifted again according to the difference between the current correction amount TS and TD.

  As shown in FIG. 9, assuming that the correction amounts of R_Gain corresponding to TS and TD are RG_TS and RG_T, respectively, for example, when the current control value and the target control value are as shown in FIG. The relationship shown is established. Therefore, it can be determined that the target color temperature is lower than the current color temperature.

[Equation 2]
RG_TS> RG_TD
In this case, the skin color determination area is shifted to the low color temperature side and is set to the position A_TD in FIG. Here, the amount by which the skin color determination area is shifted may be determined corresponding to the difference between RG_TS and RG_TD.

  As a result, P_TD1 corresponding to the skin color when the color temperature changes low falls within the A_TD region, and can be detected as the skin color.

  In the present embodiment, even when a rapid change in the color temperature of the light source that exceeds the tracking speed of white balance correction occurs, the skin color determination region corresponds to the difference between the target correction amount of the white balance control data and the current correction value. Therefore, appropriate skin color detection can be performed.

  Accordingly, similarly, the inverse orthogonal transform unit 112 can reduce the quantization step value and increase the amount of information when the image block corresponding to the face part is quantized. In addition, since the skin color determination region is subjected to timely filter processing, as a result, the code amount for the region detected as the skin color in the skin color detection processing unit 103 is relatively increased as compared with the other regions. The quantization unit 107 can be controlled.

  As a result, it is possible to accurately and stably recognize the face area of highly important people, particularly the face, without being affected by changes in the color temperature by the skin color detection process. It becomes possible to make it.

(Fourth embodiment)
In the first to third embodiments, an example has been described in which skin color is stably detected when an image is encoded, and the image quality of the region including the skin color is improved.

  However, the present invention is not necessarily effective only when an image is encoded and recorded, and is also effective when reproducing an image.

  FIG. 11 is a block diagram showing a second configuration example of an image processing apparatus typified by the video camera of the present invention.

  The operation of this embodiment when reproducing an image will be described with reference to FIG.

  In FIG. 11, a recording medium 1101 includes a memory card, a hard disk, an optical disk, and the like. A reading unit 1102 reads image data and metadata to be described later from the recording medium 1101.

  The decoding unit 1103 expands the compression-coded image data to baseband image data. The white balance detection unit 1104 detects white balance information described later from the read image data and metadata.

  The skin color detection processing unit 1105 performs the same operation as in the first to third embodiments. The filter unit 1106 also performs the same operation as in the first to third embodiments. The image output unit 1107 outputs the image to a display (not shown) or another image device. A control unit 1108 comprehensively controls the entire image processing apparatus.

  A characteristic operation of the present embodiment in the image processing apparatus having the above configuration will be described.

  As shown in FIG. 12, the recording medium 1101 in FIG. 11 records compressed and encoded image data 1201 and metadata 1202 associated therewith.

  In the metadata 1202, white balance control information at the time of shooting, color temperature information at the time of shooting, and the like are recorded.

  The read unit 1102 reads the encoded image data and metadata from the recording medium 1101. The read image data is decoded into a baseband image signal by the decoding unit 1103.

  In the present embodiment, any type of compression encoding method may be used. Further, the image recorded on the recording medium 1101 does not necessarily need to be compression-encoded. Therefore, in the present embodiment, description of the detailed configuration and operation of the decoding unit 1103 is omitted.

  The image data decoded into the baseband image by the decoding unit 1103 is input to the white balance detection unit 1104 that is a white balance detection unit. Note that the metadata read from the recording medium 1101 is also input to the white balance detection unit 1104. The decoded image data is also input to the skin color detection processing unit 1105 and the filter unit 1106.

  The filter unit 1106 is generally called a post filter because it is a filter that is processed during reproduction.

  The white balance detection unit 1104 detects the white balance control state and the color temperature of the light source from which the image is taken based on the input metadata and image data.

  FIG. 13 is a flowchart showing a procedure of white balance detection processing in the white balance detection unit in FIG.

  In FIG. 13, when the white balance detection process is started, first, metadata is read in step S1301. Next, it is determined whether or not the metadata input in step S1302 includes white balance information such as white balance control information at the time of shooting and color temperature information of the light source.

  If white balance information is included, the white balance information included in the metadata is extracted in step S1303 and output to the control unit 1108 as white balance detection data, and the process ends.

  If the white balance information is not included in the metadata in step S1303 or if the metadata itself does not exist, the image data is read in step S1304. In step S1305, a process for estimating the color temperature of the light source from the image data is performed.

  In this embodiment, the color temperature is estimated based on the average value of the color difference signals of the image. For example, when the average value of the color difference signals in the image is biased in the red direction, it is determined that the color temperature is low. Further, when the average value of the color difference signals in the image is biased in the blue direction, it is determined that the color temperature is high.

  These determination results are output to the control unit 1308 as white balance detection data in step S1306, and the process ends.

  The skin color detection processing unit 1105 performs a skin color detection operation corresponding to the white balance detection information detected by the white balance detection unit 1104, that is, the white balance operation mode and the color temperature information.

  In this embodiment as well, the skin color detection operation for changing the skin color determination region in accordance with the white balance operation mode and the color temperature information is the same as in the first to third embodiments, and thus the description thereof is omitted.

  Next, the filter unit 1106 performs a filter process on the area detected as the skin color by the skin color detection processing unit 1105.

  FIG. 14 is a flowchart showing the procedure of the filter control process of the filter unit in FIG.

  This process is performed by the control unit 1108 in FIG.

  In FIG. 14, when the filter control process is started, first, in step S1401, it is determined whether or not the color temperature is on the low color temperature side based on the output from the white balance detection unit 1104.

  If it is determined in step S1401 that the color temperature is low, the filter strength is set weak in step S1402. If it is not determined in step S1401 that the color temperature is low, it is determined in step S1403 whether the color temperature is high.

  If it is determined in step S1403 that the color temperature is higher, the filter strength is set to be strong in step S1404. If it is not determined in step S1403 that the color temperature is higher, the normal color temperature is assumed, and the filter strength is set to medium in step S1405.

  Thus, the filter strength is set stronger as the color temperature becomes higher. In the present embodiment, as the filter strength increases, an operation for further reducing high-frequency components in the image signal is performed.

  Next, in step S1406, filter processing is performed on the region determined to include skin color using the filter strength set in the above-described processing, and this flow is exited.

  Although not shown in FIG. 14, the filter processing is not particularly performed for portions other than the skin color determination region.

  The image signal subjected to these filter processes is output to the image output unit 1107 and other image devices.

  As described above, in the present embodiment, the color temperature information of the light source and the white balance control state are detected from the recorded image data, and the threshold value of the skin color determination area and the filter strength to be applied to the skin color determination area are changed as appropriate. The configuration is possible.

  In general, particularly a face portion of a person with high importance can be recognized stably with high accuracy by skin color detection processing. In addition, since the skin color determination area is timely filtered, it is possible to stably improve the image quality of the area including the skin color.

  In this embodiment, the filter strength is changed at three levels according to the color temperature. However, the filter strength may not be particularly three, and the filter strength may be changed continuously.

  The object of the present invention is achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

  Moreover, the following can be used as a storage medium for supplying the program code. For example, floppy (registered trademark) disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW, magnetic tape, nonvolatile memory card, ROM or the like. Alternatively, the program code may be downloaded via a network.

  Further, the present invention includes a case where the function of the above-described embodiment is realized by executing the program code read by the computer. In addition, based on the instruction of the program code, the O step S (operating system) running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. This is also included.

  Furthermore, a case where the functions of the above-described embodiment are realized by the following processing is also included in the present invention. That is, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 Image signal processing part 103 Skin color detection processing part 104 Filter part 105 Motion compensation prediction part 106 Orthogonal transformation part 107 Quantization part 108 Entropy encoding part 109 Buffer part 110 Recording part 111 Inverse quantization part 112 Inverse orthogonal transformation part 113 White balance operation mode selection unit 114 Control unit

Claims (10)

A color detection unit for detecting a region including a predetermined color in the image;
A filter processing unit for performing filter processing on image data;
An encoding unit for encoding image data;
White balance adjustment means for adjusting the white balance control state over a predetermined time when the color temperature of the light source at the time of shooting changes,
Determining means for determining whether or not the white balance control state adjusted by the white balance adjusting means is a low color temperature side;
Control means for changing a threshold value of a luminance signal and a color signal when detecting a predetermined color in the color detection unit according to a determination result by the determination unit ,
Wherein, when the white balance control state determines that the determination means that the low color temperature side, compared with the case where the white balance control state determines that the determination means is not the low color temperature side, the color If the threshold when detecting the skin color in the detection unit is changed in the direction of higher saturation , and the light source at the time of shooting changes during continuous shooting performed in a time shorter than the predetermined time, it is currently adjusted. depending on the difference between the control state of the control state and finally the white balance to be adjusted in white balance are, the image processing apparatus characterized that you determine the degree of change of the threshold. The control state of the white balance to be finally adjusted is obtained by calculating how much the value of the color difference signal in the area corresponding to white of the image signal obtained in this shooting is from the center of the color difference vector. The image processing apparatus according to claim 1, wherein: Wherein, when the white balance control state determines that the determination means that the low color temperature side, compared with the case where the white balance control state determines that the determination means is not the low color temperature side the image processing apparatus according to claim 1 or 2, characterized in that to reduce the code amount in image coding to be allocated to areas where skin color is detected in the coding unit. Wherein, when the white balance control state determines that the determination means that the low color temperature side, compared with the case where the white balance control state determines that the determination means is not the low color temperature side the image processing apparatus according to any one of claims 1 to 3, characterized in that weaken the strength of the filtering process performed on the region where skin color is detected in the filter processing unit. A color detection unit for detecting a region including a predetermined color from the input image data;
A filter processing unit for performing filter processing on image data;
White balance adjustment means for adjusting a white balance control state obtained from a photographed image over a predetermined time when the color temperature of the light source at the time of photographing changes ,
Determination means for determining whether or not the white balance control state adjusted by the white balance adjustment means is a low color temperature side;
Control means for changing a threshold value of a luminance signal and a color signal when detecting the predetermined color in the color detection unit according to a determination result by the determination unit ,
Wherein, when the white balance control state determines that the determination means that the low color temperature side, compared with the case where the white balance control state determines that the determination means is not the low color temperature side, the color If the threshold when detecting the skin color in the detection unit is changed in the direction of higher saturation , and the light source at the time of shooting changes during continuous shooting performed in a time shorter than the predetermined time, it is currently adjusted. depending on the difference between the control state of the control state and finally the white balance to be adjusted in white balance are, the image processing apparatus characterized that you determine the degree of change of the threshold. The control state of the white balance to be finally adjusted is obtained by calculating how much the value of the color difference signal in the area corresponding to white of the image signal obtained in this shooting is from the center of the color difference vector. The image processing apparatus according to claim 5, wherein the image processing apparatus is an image processing apparatus. When the determination unit determines that the white balance control state is on the low color temperature side , the control unit determines the flesh color in the filter processing unit as compared with the case where the determination unit determines that the white color control state is not on the low color temperature side. The image processing apparatus according to claim 5, wherein an intensity of the filter processing to be applied to a region in which the image is detected is reduced. A color detection unit for detecting a region including a predetermined color in the image;
A filter processing unit for performing filter processing on image data;
An encoding unit for encoding image data;
White balance adjustment means for adjusting the white balance control state obtained from the captured image ;
Determining means for determining whether or not the white balance control state adjusted by the white balance adjusting means is a low color temperature side;
Control means for changing a code amount at the time of image encoding assigned to an area in which the predetermined color is detected in the encoding unit according to a determination result by the determination unit ,
Wherein the control unit, when the white balance control state determines that the determination means that the low color temperature side, the photographing after the captured image, the white balance control state is not the low color temperature side An image processing apparatus characterized by reducing a code amount at the time of image encoding assigned to an area in which a flesh color is detected in the encoding unit as compared with a case where a determination unit determines . Wherein, when the white balance control state determines that the determination means that the low color temperature side, the comparison with the case where the white balance control state determines that the determination means is not the low color temperature side The image processing apparatus according to claim 8 , wherein an intensity of the filter processing applied to an area in which a skin color is detected in the filter processing unit is reduced. A color detection unit for detecting a region including a predetermined color in the image;
A filter processing unit for performing filter processing on image data;
An encoding unit for encoding image data;
White balance adjustment means for adjusting the white balance control state obtained from the captured image ;
Determining means for determining whether or not the white balance control state adjusted by the white balance adjusting means is a low color temperature side;
Control means for changing the type and intensity of the filter processing to be performed on the region where the predetermined color is detected in the filter processing unit according to the determination result by the determination unit ,
Wherein the control unit, when the white balance control state determines that the determination means that the low color temperature side, the photographing after the captured image, the white balance control state is not the low color temperature side An image processing apparatus characterized in that the intensity of the filter processing to be performed on an area in which the skin color is detected in the filter processing unit is reduced as compared with the case where the determination means determines .

Images