JP2017152866A - Image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a natural correction result by adequately determining a state of shade to be used as a correction target, when a rewriting correction is made to an image including a plurality of subjects.SOLUTION: A subject in an input image is detected, the contrast value of each subject is detected, and a subject the detected contrast value of which is largest and a subject the detected contrast value of which is smallest are selected as subjects to be corrected. A target contrast value is set so that the subject the selected contrast value of which is largest is decreased in contrast value and the subject the contrast value of which is smallest is increased in contrast value. A virtual light-source is set so that the contrast value of a subject to be corrected is equal to the target contrast value. Using the virtual light-source, the brightness of the subject to be corrected is corrected.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus and method for correcting the brightness of an input image.

2. Description of the Related Art Conventionally, a technique for performing relighting by irradiating light from a virtual light source (virtual light source) to a subject in an image after shooting is known. Thereby, the shadow of the subject caused by the ambient light can be corrected after shooting.
For example, Patent Document 1 discloses a relighting processing unit that performs pseudo lighting processing on a captured image. Specifically, with respect to a method for determining the position of a virtual light source, a technique for correcting the subject to reduce the shadow of the subject by disposing the ambient light and the virtual light source in opposite directions is disclosed.

JP 2010-135996 A

  By performing the relighting correction, it is possible to correct the shadow state of the subject so as to approach the target shadow state. As the shadow state targeted for relighting correction, when there is only one subject in the image, a predetermined shade state that is desirable to the eye may be determined.

  On the other hand, when there are a plurality of subjects in the image, it is necessary to determine the target shade state in consideration of the difference in the shade state between the subjects, instead of simply determining the preferred shade state to the eye. is there. For example, the difference in the state of shadow between subjects such as the magnitude relationship of the contrast reflects the difference in how ambient light strikes each subject when shooting. For this reason, if the relationship of the shadow state between the subjects is reversed before and after the relighting correction, it is considered that there may be an unnatural image that contradicts the way the ambient light hits at the time of shooting. .

  However, the conventional technique described in Patent Document 1 does not disclose a method for setting a target shadow state when an image includes a plurality of subjects, and appropriate relighting correction for a plurality of subjects is performed. There was a case that could not be controlled.

  Accordingly, an object of the present invention is to determine a natural shadow state as a target for relighting correction when performing relighting processing using a virtual light source on an image having a plurality of subjects. It is an object to provide an image processing apparatus and method capable of obtaining a relighting result.

  In order to achieve the above object, the present invention has the following configuration.

Subject detection means for detecting a subject in the image of interest;
State detecting means for detecting a shadow state indicating a degree of shadow for each subject;
Based on the detected shadow state, a target for each target subject in the image of interest is arranged so that the order of the degree of shadow before correction of the subject and the order of shadow after correction are not interchanged. Target setting means for setting a target shadow state that is a shadow state to become,
A light source setting unit that determines a setting of a virtual light source for each target subject based on the shadow state detected by the state detection unit and the target shadow state for the target image;
An image processing apparatus comprising: a correction unit that corrects the brightness of the subject subject in the image of interest using the virtual light source.

  According to the present invention, when performing relighting processing on an image in which a plurality of subjects exist, a natural relighting result can be obtained by appropriately setting a shadow state targeted for relighting correction for each subject. It becomes possible.

It is a block diagram which shows the structure of a digital camera. It is a block diagram which shows the structure of an image process part. It is a flowchart which shows the flow of the whole relighting process in 1st Example of this invention. It is a figure which shows the example of the to-be-photographed object in 1st Example of this invention. It is a figure explaining the setting method of the target shadow state in 1st Example of this invention. It is a figure which shows the example of the method of determining the position of the virtual light source in the 1st Example of this invention. It is a figure which shows the example of the positional relationship of a virtual light source and a to-be-photographed object in 1st Example of this invention. It is a flowchart which shows the flow of the setting process of the target shadow state in 2nd Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, an example in which the image processing apparatus is applied to a digital camera will be described.

A digital camera according to the first embodiment of the present invention will be described below with reference to FIGS.
<Configuration of digital camera>
FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to an embodiment of the present invention. The present invention is characterized by an image processing applied to a captured image, more specifically, a relighting processing method. Therefore, a configuration relating to shooting and recording is not essential.

  In FIG. 1, a lens group 101 is a zoom lens including a focus lens. A shutter 102 having a diaphragm function is provided between the lens group 101 and the imaging unit 103. The imaging unit 103 includes an imaging element represented by a CCD / CMOS image sensor that converts an optical image formed on the imaging surface by the lens group 101 into an electrical signal in pixel units. The A / D converter 104 converts the analog signal output from the imaging unit 103 into a digital signal (image data).

  The image processing unit 105 performs various image processing such as color interpolation (demosaic), white balance adjustment, and γ correction on the image data output from the A / D converter 104. The image processing unit 105 also performs relighting processing on the captured image. The image memory 106 temporarily stores image data. A memory control unit 107 controls reading and writing of the image memory 106. The D / A converter 108 converts the image data into an analog signal. The display unit 109 includes a display device such as an LCD or an organic EL display, and displays various GUIs, live view images, images read out from the recording medium 112 and reproduced. The codec unit 110 encodes the image data stored in the image memory 106 by a predetermined method for recording on the recording medium, or decodes the encoded image data included in the image file for display, for example. Or

  The interface (I / F) 111 mechanically and electrically connects a detachable recording medium 112 such as a semiconductor memory card or a card type hard disk to the digital camera 100. The system control unit 50 may be a programmable processor such as a CPU or MPU. The system control unit 50 implements the functions of the digital camera 100 by executing a program recorded in the nonvolatile memory 121 and controlling necessary blocks and circuits.

  The face detection unit 113 detects a face area included in the photographed image, and obtains face information such as a position, a size, and reliability for each of the detected face areas. Note that the face detection unit 113 uses a technique such as learning represented by a neural network, a feature part such as an eye, nose, or mouth is searched from an image area using template matching, and is considered as a face if the similarity is high. The face area can be detected by using the method.

  The operation unit 120 collectively describes input devices such as buttons and switches for the user to input various instructions to the digital camera 100. When the display unit 109 is a touch display, the touch panel is included in the operation unit 120. Further, the operation unit 120 may include an input device of a type that inputs an instruction in a non-contact manner such as voice input or line-of-sight input.

  The nonvolatile memory 121 may be an electrically erasable / recordable, for example, an EEPROM. The nonvolatile memory 121 stores various setting values and GUI data, and programs to be executed by the system control unit 50 when the system control unit 50 is an MPU or CPU. The system memory 122 is used to develop constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like.

<Basic operation of digital camera>
Next, a basic operation at the time of photographing an object in the digital camera 100 configured as described above will be described. The imaging unit 103 photoelectrically converts the subject image formed on the imaging surface by the lens group 101 when the shutter 102 is opened, and outputs the subject image to the A / D converter 104 as an analog image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.
The image processing unit 105 performs various image processing such as synchronization processing (demosaic processing), γ correction, and relighting processing on the image data from the A / D converter 104 or the image data from the memory control unit 107. . Further, the image processing unit 105 performs predetermined calculation processing relating to brightness, contrast, and the like using image data obtained by photographing, and the system control unit 50 performs focus adjustment and exposure control based on the obtained calculation result. . You may consider the detection result of the face detection part 113 for focus detection or exposure control. As described above, the digital camera 100 according to the present embodiment performs TTL (through the lens) AF (autofocus) processing and AE (automatic exposure) processing. The image processing unit 105 further performs auto white balance (AWB) adjustment using image data obtained by shooting.

The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 109.
The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display in accordance with an analog signal from the D / A converter 108 on a display device such as an LCD.
The codec unit 110 encodes the image data recorded in the image memory 106 based on a standard such as JPEG or MPEG. The system control unit 50 assigns a predetermined header or the like to the encoded image data, forms an image file, and records the image file on the recording medium 112 via the interface 111.

  Note that with current digital cameras, it is common to perform moving image shooting in the shooting standby state, and keep the captured moving image displayed on the display unit 109 so that the display unit 109 functions as an electronic viewfinder (EVF). is there. In this case, the shutter 102 is opened and a so-called electronic shutter of the imaging unit 103 is used to capture an image at, for example, 30 frames / second.

  When the shutter button included in the operation unit 120 is half-pressed, the above-described AF and AE control is performed. When the shutter button is fully pressed, the still image shooting for recording is performed by the main shooting and is recorded on the recording medium 112. The Also, when moving image shooting is instructed by a moving image shooting button or the like, moving image recording on the recording medium 112 is started.

<Example of functional configuration related to relighting>
FIG. 2 is a block diagram illustrating a functional configuration example of the image processing unit 105 related to the relighting process. Note that one or more of the functional blocks shown in FIG. 2 may be realized by a combination of a microprocessor and software, or may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device). May be. PLD includes FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array), and the like. Note that the relighting process is performed on an image captured in a state where execution of the relighting process is specified, or an image recorded on the recording medium 112, for example, on which execution of the relighting process is instructed from a menu screen or the like. Can do. When information at the time of shooting is necessary in the relighting process, the information is read from the nonvolatile memory 121 or the system memory 122 or acquired from the header of the image file. In FIG. 2, the image processing unit 105 includes an image signal generation unit 201, a WB amplification unit 202, a subject area extraction unit 203, a shadow state detection unit 204, a target shadow state setting unit 205, a virtual light source setting unit 206, and a virtual light source signal addition. A unit 207 and a gamma processing unit 208.

<Relighting process>
Next, the flow of processing in the image processing unit 105 will be described. FIG. 3 is a flowchart showing the overall flow of relighting processing executed by each unit of the image processing unit 105.
In step S <b> 301, the subject region extraction unit 203 extracts only the image signal corresponding to the subject region from the image signals input from the WB amplification unit 202. That is, subject detection is performed. The input image signal is referred to as an input image or a target image. For example, when the subject area to be rewritten is a human face area, an image signal corresponding to the face area of the subject person detected by the face detection unit 113 is extracted. When there are a plurality of subjects, a plurality of subject regions are extracted.
In step S302, the subject area extraction unit 203 determines whether a plurality of extracted subject areas exist in the image of interest. If it is determined that there are a plurality of subject areas, the process proceeds to step S303, and if the subject area is single, the process proceeds to step S307.

In step S303, the shadow state detection unit 204 detects the shadow state of the subject for each of the extracted subject regions. The shadow state detection unit 204 outputs information representing the detected shadow state of the subject to the target shadow state setting unit 205. For example, the state of shading refers to the distribution of brightness in the subject area, information indicating the contrast in the subject area, the characteristics of the contour of the shaded area in the subject area, and the shadow in the subject area. Including at least one piece of information representing the area of the area. In this embodiment, the contrast value is mainly used as an evaluation value indicating the shadow state.
In step S <b> 304, the target shadow state setting unit 205 determines a subject for relighting correction based on the comparison result of the information representing the shadow state of each input subject.
In step S305, the target shadow state setting unit 205 sets a target based on the result of comparing the information representing the input shadow state of the subject for each of the subjects selected in step S304. That is, the target shadow state, which is the target shadow state for relighting correction, is determined. The target shadow state setting unit 205 outputs information representing the determined target shadow state to the virtual light source setting unit 206.
In step S306, the virtual light source setting unit 206 determines each of the subjects selected in step S304 based on the input information indicating the target shadow state and the information indicating the shadow state of the subject detected by the shadow state detection unit 204. On the other hand, the set value of the virtual light source is determined. The virtual light source setting unit 206 outputs the determined virtual light source setting value to the virtual light source signal adding unit 207.

In step S307, the shadow state detection unit 204 detects the shadow state of the subject with respect to the extracted subject region. The shadow state detection unit 204 outputs information representing the detected shadow state of the subject to the target shadow state setting unit 205.
In step S308, the target shadow state setting unit 205 determines a target shadow state, which is a target shadow state for relighting correction, based on the input information indicating the shadow state of the subject. However, when the process proceeds to step S308, only one subject to be corrected is detected, and it is not necessary to determine the target shadow state by comparing the shadow states between the subjects. For this reason, here, a predetermined shadow state determined in advance is determined as the target shadow state. The target shadow state detection unit 205 outputs information representing the determined target shadow state to the virtual light source setting unit 206.
In step S309, the virtual light source setting unit 206 determines a setting value of the virtual light source based on the input information indicating the shadow state of the subject and the information indicating the target shadow state. The virtual light source setting unit 206 outputs the determined virtual light source setting value to the virtual light source signal adding unit 207. Here, the method for determining the setting value of the virtual light source is the same as the method used in step S306.

In step S <b> 310, the virtual light source signal adding unit 207 generates a signal corresponding to the virtual light source based on the input setting value of the virtual light source, and adds the signal to the image signal input from the WB amplification unit 202. If a plurality of subjects to be relighted are selected, a signal corresponding to the virtual light source is generated based on the setting value of the virtual light source determined in step S306 for each subject.
The above is the overall flow of the relighting process in the present embodiment. Hereinafter, details of processing in each step will be described in order.

<Detection process of shadow state of subject>
Details of processing in which the shadow state detection unit 204 detects the shadow state of the subject caused by ambient light will be described with reference to FIG. This process corresponds to steps S303 and S307 in FIG. The shadow state detection unit 204 calculates the brightness information of the subject based on the image signal of the subject area input from the subject region extraction unit 203. Specifically, as shown in FIG. 4, the subject area is divided into a plurality of parts (for example, 8 × 8 = 64 blocks), and an average luminance value is calculated for each of the divided blocks. Next, the shadow state detection unit 204 calculates the contrast value of the subject based on the calculated luminance average value. Specifically, the block having the highest average luminance (402) and the block having the lowest average luminance (403) are selected from the plurality of divided blocks, and the ratio of the average luminance is calculated and used as the contrast value. And However, in addition to the luminance signal, a color signal such as hue and saturation is calculated for each block, and a contrast value is calculated between blocks having close subject colors. Whether the subject colors are close is determined by, for example, the difference between the values of color signals such as hue and saturation being within a predetermined value. When calculating the contrast value between blocks with close subject colors, for example, the blocks of the subject are classified into groups with close subject colors, and the contrast value is calculated within each group. The largest contrast value between the groups is set as the final contrast value. The shadow state detection unit 204 outputs the calculated contrast value to the target shadow state setting unit 205 as subject shadow information.

  The shadow state detection unit 204 estimates the position of the ambient light. Various known methods can be used to estimate the position of the ambient light. For example, the brightest area in the subject may be detected, the normal vector of the area may be calculated, and the direction may be determined as the direction in which the ambient light is located. Further, in the case of a human face region, the direction of the ambient light may be estimated by detecting an organ such as the nose and detecting the shadow position. The shadow state detection unit 204 outputs the estimated position information of the ambient light to the virtual light source setting unit 206.

<Relighting target decision process>
Next, the process in which the target shadow state setting unit 205 determines a subject to be rewritten based on the subject shadow information will be described in detail. This process corresponds to step S304 in FIG.
FIG. 5 shows an example of the result of calculating the contrast value of each subject when the image includes a plurality of subjects. In the figure, the horizontal axis represents subject distinction, and the vertical axis represents the contrast value. Ca to Cd in the figure represent the contrast values of the subjects a to d, respectively. As shown in the figure, there is a relationship of Ca <Cb <Cc <Cd.

  The target shadow state setting unit 205 selects the subject with the lowest contrast value and the subject with the highest contrast value as the relighting correction target. In the case shown in FIG. 5, subjects a and d are subject to relighting correction, and subjects b and c are not subject to relighting correction. In this way, it is possible to set only the subject that is considered highly necessary to correct the shadow state by the relighting correction as the target of the relighting correction. On the other hand, since relighting correction is not performed on other subjects, it is possible to generate an image that retains the shaded state due to ambient light at the time of shooting.

<Determination processing of target shadow state of subject>
Next, the details of the process in which the target shadow state setting unit 205 determines the target shadow state for the subject selected as the relighting correction target will be described. This process corresponds to steps S305 and S308 in FIG. Here, the target shadow state is determined so that the contrast value of the subject approaches the reference contrast value. Further, as a characteristic process in the present embodiment, the target shadow state for each subject is determined so that the magnitude relationship of the contrast value of the subject does not reverse before and after the relighting correction. In other words, for each subject to be relighted, the target and the target are rewritten so that the order of the shade of each subject represented by the shadow state before the correction and the order of the shade of each subject after the correction are not interchanged. The target shadow state that is the state of the shadow to be set is set. By doing so, it is possible to control relighting correction so as to make it appear close to the appearance of a shade that is desirable to the eye and to achieve a natural appearance that leaves the influence of the original ambient light.

  Specifically, processing is performed as follows. First, the target shadow state setting unit 205 calculates an average value of subject contrast values as a reference contrast value. An example of the calculated average value is shown in Cave in FIG. The target shadow state setting unit 205 determines a target contrast value so that the subject contrast value is higher (lower) if the subject contrast value is lower (higher) than Cave. For example, in the case of the subject a, since Ca <Cave, the target shadow state is set so that the subject contrast value becomes high. However, Ca ′ = Cb−ΔC, which is a value lower than the contrast value Cb of the subject b by a predetermined value ΔC, is set as the target shadow state so that the corrected contrast value of the subject a does not become higher than the contrast value Cb of the subject b. decide. Here, ΔC is a constant having a positive value, and Cb−Ca ≧ ΔC> 0 (condition 1). Alternatively, ΔC may be set as the target shadow state. In this case, the target shadow state is the amount of contrast correction.

  Similarly, in the case of the subject d, since Cave <Cd, the target shadow state is set so that the subject contrast value becomes low. However, Cd ′ = Cc + ΔC is determined as the target shadow state so that the corrected contrast value of the subject d does not become lower than the contrast value Cc of the subject c. That is, Cd−Cc ≧ ΔC> 0 (condition 2). A value satisfying both condition 1 and condition 2 is selected as ΔC. The target shadow state setting unit 205 determines a target contrast value for each subject by the above method, and outputs the result as a target shadow state to the virtual light source setting unit 206.

<Virtual light source setting value determination process>
Next, the details of the process in which the virtual light source setting unit 206 calculates the setting value of the virtual light source based on the subject shadow information and the target shadow state of each subject will be described. This process corresponds to steps S306 and S309 in FIG. FIG. 6 is a flowchart showing a flow of processing performed by the virtual light source setting unit 206. FIG. 6 shows a processing procedure for one subject. When there are a plurality of subjects, the processing procedure of FIG. 6 is performed on the subject of interest while paying attention to each subject sequentially. FIG. 7 shows the positional relationship among the subject, the ambient light, and the virtual light source. Here, a case where the correction target area is a human face as shown in FIG. 4 will be described.

  FIG. 7 shows a state seen from above the subject. The shaded portion of the subject region 701 represents, for example, a human face region, and the surface is represented by a plane for simplicity. A normal vector 702 is a typical normal vector representing the direction of the subject area 701. Various known methods can be used as representative normal vector calculation methods. For example, the normal vector can be calculated by estimating the three-dimensional shape of the subject from the result of measuring the distance to the subject by a distance measuring unit (not shown). In the case of a human face, the normal vector can be estimated based on the result of estimating the face direction by detecting the face and organs such as eyes and nose by the face detection unit 113. . The representative direction of the normal vector of the subject can be calculated by averaging the calculated normal vector over the entire subject area. Alternatively, a normal vector at the center position of the subject area to be relighted may be used as a representative normal vector.

  The ambient light 703 irradiates the coordinates (pixels) 710 where the representative normal vector 702 and the face region 701 intersect from the direction estimated by the shadow state detection unit 204. Virtual lights 704a and 704b are images of virtual light that irradiates coordinates 710 from respective positions. Here, an angle formed by the normal vector 702 and the irradiation direction of the virtual lights 704a and 704b is expressed by an angle α, and an angle formed by the normal vector 702 and the irradiation direction of the ambient light 703 is expressed by an angle β. The angle α is also the angle formed by the normal vector 702 and the straight line connecting the coordinates of the virtual light sources 704a and 704b and the start point of the normal vector 702 (that is, the intersection 710 of the normal vector 702 and the subject area 701). I can say that. Similarly, the angle β can also be said to be an angle formed by the normal vector 702 and a straight line connecting the coordinates of the environment light source and the starting point of the normal vector 702 (intersection of the normal vector 702 and the subject area 701). In addition, the angle formed by the normal vector 702 and the irradiation directions of the virtual light sources 704a and 704b and the ambient light 703 takes a positive sign counterclockwise and a negative sign clockwise. Therefore, for example, if the angle of the normal vector 702 is set as the reference (that is, 0) with the coordinate 710 as the origin, the angle formed by the normal vector 702 and the irradiation direction of the ambient light 704b can be expressed as -α. Further, the angle formed by the normal vector 702 and the irradiation direction of the ambient light 704a can be expressed as + α.

  Note that the direction of the light source described here is a direction on a plane as illustrated in FIG. 7, but this direction is orthogonal to the subject plane and orthogonal to each other when the subject is considered as a plane, for example. If determined in two planes, the direction of the light source in the three-dimensional space can be determined. For example, in the three-dimensional orthogonal coordinate system, when the intersection 710 is the origin and the subject plane is the YZ plane, the direction of the light source is determined on each of the XY plane and the ZX plane, thereby providing a light source in the three-dimensional space. The direction can be determined.

<Virtual light source setting value determination processing procedure>
The flow of processing will be described using the flowchart of FIG. Here, a case where the virtual light source setting unit 206 calculates the position of the virtual light source (light irradiation direction) as the setting value of the virtual light source will be described. Therefore, the execution subject of each step in FIG. 6 is the virtual light source setting unit 206.
In step S <b> 601, the target contrast value information determined in step S <b> 305 is acquired from the target shadow state setting unit 205.
In step S <b> 602, the subject contrast value generated by the ambient light detected in step S <b> 303 is acquired from the shadow state detection unit 204. In step S303, the position of the ambient light is also detected by the shadow state detection unit 204, and the position information of the ambient light with respect to the subject is acquired together with the subject contrast value.
In step S603, the target contrast value is compared with the subject contrast value to determine whether the subject contrast value is higher than the target contrast value. If it is determined that the subject contrast value is higher than the target contrast value, the process proceeds to step S604; otherwise, the process proceeds to step S605.
The case where the process proceeds to step S604 corresponds to a case where a dark shadow is generated in the subject area due to the ambient light and the subject contrast value is higher than the target contrast value. For this reason, in step S604, the virtual light source setting unit 206 calculates the position (direction) of the virtual light source so that the subject is irradiated from different sides of the virtual light source and the ambient light. The side different from the ambient light indicates, for example, the side opposite to the ambient light source across the normal of the subject. Specifically, the irradiation direction of the virtual light is determined so that the sign of the angle α is opposite to the sign of the angle β and the absolute value of the angle α is 45 °.

  Here, the reason why the absolute value of the angle α is set to 45 ° is that an appropriate contrast is often obtained when a normal subject is irradiated from an angle of 45 ° with respect to the normal vector 602. However, this is merely an example, and the present invention is not limited to this as long as the shadow portion of the subject can be brightly corrected with virtual light to reduce the contrast of the subject region. In the example of FIG. 7, the virtual light 704b is set in step S604. Here, the irradiation direction of the virtual light 704b may be a direction (−β) that is symmetric with the ambient light 703 with the normal vector 702 as an axis of symmetry. Here, the angle of the normal vector 702 is set to zero. As another example, it may be set within a range of 45 ° ± 10 °, preferably 45 ° ± 5 °. The subject contrast value after the relighting process may be calculated, and the irradiation direction may be determined so that the value is closest to the target contrast value. When the irradiation direction of the virtual light is calculated, the virtual light source setting unit 206 advances the process to S608.

On the other hand, in step S605, it is determined whether the subject contrast value is equal to or less than a predetermined threshold value. When it determines with it being below a predetermined threshold value, it progresses to step S606, and when that is not right, it progresses to step S607.
The case where the process proceeds to step S606 corresponds to a case where the subject contrast value is lower than the target contrast value, and the ambient light is irradiated almost uniformly on the subject, and there is almost no shadow due to the ambient light. . In this case, the virtual light source setting unit 206 determines the irradiation direction of the virtual light so that the absolute value of the angle α formed by the irradiation direction of the virtual light with the normal vector 702 is 45 ° in step S606. Since the sign of the angle α may be positive or negative, the angle α has two values of 45 ° and −45 °. However, setting the absolute value of the angle α to 45 ° is an example of a preferable value, and is not limited to this as long as a part of the subject area can be brightly corrected by the virtual light source and the contrast of the subject area can be increased. For example, the subject contrast value after the relighting process may be calculated, and the irradiation direction may be determined so that the value is closest to the target contrast value. When the irradiation direction of the virtual light is calculated, the virtual light source setting unit 206 advances the process to S608.

  The process proceeds to S607 when the subject is shaded to some extent by the ambient light, but the target contrast is not obtained. Therefore, in step S607, the virtual light source setting unit 206 has the same sign as the angle β that the illumination direction of the virtual light makes with the normal vector 702 and the angle β that makes the illumination direction of the ambient light the normal vector 702, The irradiation direction of the virtual light is determined so that the absolute value of the angle α is 45 °. However, setting the absolute value of the angle α to 45 ° is an example of a preferable value, and is not limited to this as long as a part of the subject area can be brightly corrected by the virtual light source and the contrast of the subject area can be increased. For example, the absolute value of the angle α may be calculated based on the absolute value of the angle β. Specifically, the absolute value of the angle α may be | β | ± 15 °, preferably | β | ± 10 °, more preferably | β | ± 5 °. Here, ± may indicate a range. Alternatively, α = β, that is, the same irradiation direction as the ambient light may be determined as the virtual light irradiation direction. Alternatively, the subject contrast value after the relighting process may be calculated, and the irradiation direction may be determined so that the value is closest to the target contrast value. When the irradiation direction of the virtual light is determined, the virtual light source setting unit 206 advances the process to S608.

  In step S608, the calculated arrangement of the virtual light source is output to the virtual light source signal adding unit 207 as a setting value of the position of the virtual light source. The correction amount (ΔC) for the subject a and the correction amount (ΔC) for the subject d are not necessarily the same.

  As described above, the virtual light source setting unit 206 calculates the setting value of the virtual light source corresponding to the shadow state for each correction target region, and outputs it to the virtual light source signal adding unit 207. The distance D from the intersection point 710 to the light source may be given as a constant, for example. Settings such as the direction and intensity of the virtual light source are determined so that the magnitude relationship (or strength relationship) between the shadow state before correction of the subject by relighting, which will be described later, and the shadow state after correction is not reversed. That is, for example, referring to the example of FIG. 5, the contrast, that is, the correction amount ΔC of the shadow state satisfying Cb−Ca ≧ ΔC> 0 (condition 1) and Cd−Cc ≧ ΔC> 0 (condition 2) is obtained. Thus, the setting of the virtual light source is determined. For this purpose, for example, assuming that the brightness of the brightest area of the subject does not change depending on the virtual light source, the target brightness of the darkest area of the subject is calculated to obtain the target contrast value. And the intensity | strength and distance of the virtual light source for correct | amending the brightness | luminance before correction | amendment of the darkest area | region to target brightness | luminance are determined according to Numerical formula 1 mentioned later. At this time, the direction of the virtual light source (angle with respect to the normal) is determined. Here, it is assumed that the brightness of the brightest area of the subject is not changed by the virtual light source, but the brightness of this area may also be increased by the virtual light source. In this case, since the corrected contrast value may be larger than the target contrast value, the predetermined value may be subtracted from the determined intensity or distance of the virtual light source.

<Relighting process using virtual light source>
Next, relighting processing using a virtual light source in the virtual light source signal adding unit 207 will be described. This process corresponds to step S310 in FIG. In the present embodiment, the output RGB values (Rout, Gout, Bout) of the processing target pixels irradiated by the virtual light source are calculated by the following equations.

Rout = [Rt + A × cos (α) × (1 / D ^ 2) × Rv] / M
Gout = [Gt + A × cos (α) × (1 / D ^ 2) × Gv] / M (Formula 1)
Bout = [Bt + A × cos (α) × (1 / D ^ 2) × Bv] / M
However, if | α | ≦ 90 ° and | α |> 90 °, the light is not illuminated by the light from the virtual light source, so the output value remains the pixel value to be processed. (Rt, Gt, Bt) is a pixel value to be processed, A is a predetermined constant representing the intensity of the virtual light source (that is, the intensity of virtual illumination light emitted from the virtual light source), D is the distance between the virtual light source and the subject, (Rv, Gv, Bv) represents the light source reflection color. The light source reflection color is a color obtained by estimating the reflection color when the virtual light source is reflected on the subject surface by the preset virtual light source color and subject color. M is a preset constant for normalizing the output RGB value after relighting. The angle α is an angle formed by the direction of the virtual light source determined by the virtual light source setting unit 206 and the normal vector of the subject in the target pixel. In Equation 1, A × cos (α) × (1 / D ^ 2) is the intensity of the virtual light source on the subject surface, and the value is calculated from the virtual light source out of the light intensity from the virtual light source on the subject surface. The intensity component perpendicular to the subject surface of the light is calculated as being inversely proportional to the distance of the virtual light source. Then, the calculated value is multiplied by the light source reflection color and added to the pixel value, and the normalized value is calculated as the pixel value of the target pixel.

  Here, when calculating the angle α, the normal α of the subject may be calculated for each pixel of the relighting target pixel to calculate the angle α. Alternatively, as shown in FIG. 4, the subject region may be divided into a plurality of blocks, and the angle α may be calculated based on the normal vector calculated for each block. Alternatively, when the three-dimensional shape of the subject is known in advance, such as a human face, the direction of the normal vector is estimated based on where the target pixel is in the face region, and the angle α is set to It may be calculated. The intensity A of the virtual light source may be determined according to the difference ΔC or −ΔC between the subject contrast value and the target contrast. The distance D of the virtual light source may be a predetermined distance. If the distance D is a constant distance, 1 / D ^ 2 is a constant, so A * (1 / D ^ 2) can also be given as a constant. In this case, the value of A × (1 / D ^ 2) may be determined according to the difference | ΔC | between the subject contrast value and the target contrast (this is called the contrast adjustment amount or the shadow state adjustment amount). For example, the contrast value is calculated for each subject from the pixel value changed by Equation 1 in the manner of step S303, and A × (1 / D ^ 2) and the constant M are determined so that the contrast value becomes the target contrast value. A × (1 / D ^ 2) may be determined in advance and adjusted by a constant M, or vice versa.

The configuration of this embodiment is summarized as follows.
1) Detect a subject in the input image.
2) The contrast value of each subject is detected.
3) Select the subject with the largest detected contrast value and the subject with the smallest contrast as the subject to be corrected, lower the contrast value for the subject with the largest selected contrast value, and raise the contrast value for the subject with the smallest contrast value. The target contrast value is set as follows. At this time, the target contrast value is determined so that the order of the contrast values of the subjects in the input image is maintained.
4) The virtual light source is set so that the contrast value of the subject to be corrected becomes the target contrast value.
5) The brightness of the subject to be corrected is corrected using a virtual light source.

  As described above, in this embodiment, when relighting correction is performed on a plurality of subject areas, based on the result of comparing the shadow states of the subjects, the target shadow state that is the target of relighting correction for each subject. Was controlled to determine. As a result, when relighting correction is performed on an image including a plurality of subjects, it is possible to prevent the shadowing state between subjects from being reversed and resulting in an unnatural image due to the relighting correction.

[Modification 1]
In this embodiment, the case where the shadow state is detected for all of the plurality of detected subjects has been described. However, the present invention is limited to the method for detecting the shadow state in the relighting process. Not what you want. For example, only a subject whose area in the image occupies a predetermined threshold or more may be selected from a plurality of detected subjects, and the shadow state may be detected. Or you may detect the characteristic of the outline of the area | region where the shadow has arisen among the to-be-photographed objects. For example, if the subject is a person's face, the front face and the side face are discriminated based on the contour, and only those subjects whose face direction is close to the front are selected, and subjects that are close to the side are excluded. The shadow state may be detected.

[Modification 2]
Further, in this embodiment, the description has been given of the case where the subject having the lowest (highest) contrast value among the plurality of subjects is the target of the relighting correction. However, the present invention provides a method for determining the target of the relighting correction. It is not limited. That is, any method may be used as long as it is a method for determining the relighting correction target by comparing the shaded states of a plurality of subjects. For example, the subject may be selected as a relighting correction target when the difference between the reference value and the subject contrast value exceeds a predetermined magnitude compared with a predetermined reference value. In addition, among a plurality of subjects, a subject whose contrast value is lower (higher) than the average value of the plurality of subjects may be selected as a relighting correction target. In addition, among a plurality of subjects, a subject having a contrast value lower (higher) than the subject contrast value of the main subject may be selected as a relighting correction target. The most main subject may be, for example, the subject with the largest area, the detected face, or the subject in focus. The degree of focus can be determined by, for example, edge extraction. Further, subjects close to each other in the image may be compared, and a subject having a low (high) contrast value may be selected as a relighting correction target.

  Also, subjects whose shadows are significantly different from those of other subjects, that is, subjects whose contrast value is different from the contrast value of other subjects by a predetermined threshold or more, are excluded from the relighting correction target. You may control. By doing so, the shaded state is greatly different from that of other subjects, so that when relighting correction is performed, the correction amount becomes large, and control can be performed so as not to cause a harmful effect such that the noise component is emphasized.

[Modification 3]
Further, in the present embodiment, the case where the average value of a plurality of subject contrast values is used as a reference when determining the target shadow state has been described, but the target value is based on the result of comparing the shadow states of the subject. Any method may be used as a reference as long as it is a method for determining the value. For example, the target contrast value for each subject may be determined using a predetermined contrast value as a reference. Further, the contrast value of the most main subject among a plurality of subjects may be used as a reference.

[Modification 4]
In the present embodiment, the target shadow state is set by using the relationship between the corrected subject contrast value of the subject selected as the relighting target and the subject contrast value of the subject other than the relighting target. The method for setting the target shadow state is not limited to this. For example, when a plurality of subjects are selected as relighting targets, the target shadow state may be set so that the magnitude relationship between the subject contrast values before correction is maintained even after correction.

[Modification 5]
In the present embodiment, the target value for relighting correction is set as the contrast value of the subject and the magnitude relationship with the contrast value with other subjects is set so as not to reverse, but the target value for relighting correction is determined. The method is not limited to this. In other words, any method may be used as long as it is a method for determining a target value for relighting correction using a result obtained by comparing shade states of a plurality of subjects. For example, the position of the shadow area of the subject may be determined as a relighting correction target. Then, for a plurality of subjects, the position of the shadow area in each subject is determined (for example, if the subject is a human face, which side of the face is the shadow region) A target value for relighting correction is determined so as not to differ from other subjects. Further, information on the brightness of the subject may be used as information representing the state of the shadow. Specifically, for multiple subjects, the average brightness in each subject area is calculated, and the target value for relighting correction is set so that the brightness relationship between subjects does not reverse before and after relighting correction. It may be determined.

[Modification 6]
Alternatively, a shadow area of the subject may be detected, and information representing the boundary characteristics may be used as information representing the shadow state of the subject. Specifically, the brightness gradient of the boundary of the shadow area of the subject is calculated, and the position of the virtual light source and the diffusion characteristics of the virtual illumination light so that the value is comparable to that of other subjects May be determined. Also, the area of the shadow area in the subject area is calculated, the result is used as information representing the shadow state of the subject, and the relighting correction is controlled based on the result of comparing the value with other subjects. May be.

[Modification 7]
In this embodiment, the case where the target shadow state is separately determined for each of the plurality of subjects has been described. However, the present invention limits the method of determining the target shadow state to this. Not what you want. That is, any method may be used as long as it compares the shadow states of a plurality of subjects and determines the target shadow state for each subject. For example, by comparing the shadow conditions of multiple subjects, the method of estimating ambient light is estimated, and subjects with similar environmental light exposure are grouped together, and the target shadow status is determined for each group. You may do it.

[Modification 8]
Further, in this embodiment, the case where the position of the virtual light source is controlled as the setting of the virtual light source has been described. However, the present invention is not limited to this method of controlling the setting of the virtual light source. For example, the parameters of the intensity, color, number, and shape of the virtual light source may be controlled to control the relighting correction so that the shadow state of the subject approaches the target shadow state.

[Modification 9]
In Formula 1 of the present embodiment, if the value of the constant M is too far from 1, the difference between the pixel values of neighboring pixels that originally have approximate colors may be enlarged and a boundary line may be generated. For example, a pixel value corrected by a virtual light source in the vicinity of α = ± 90 ° (for example, Rout≈Rt / M because cos α≈0), and a pixel value that is in the vicinity and is not subject to correction because | α |> 90 °. The difference from (for example, Rout = Rt) can be large while being a neighboring pixel. As a result, a boundary line that should not originally exist in the image may appear. Therefore, for example, only the correction value (or correction amount) added to the original pixel value is divided by the normalization constant M. For example, in the case of the R component, Rout = Rt + [A × cos (α) × (1 / D 2) × Rv] / M or the like.

[Modification 10]
Further, for example, one virtual light source may be provided for each subject, and correction may be performed as follows. For example, the position of the center of gravity is determined as a representative pixel for each subject area, and the normal vector is set as the representative normal vector of the subject. The position (direction) of the virtual light source of the subject is set with respect to the representative normal vector. Here, the direction of the virtual light source with respect to the representative normal vector is represented by an angle α0. Then, the angle αp of the virtual light source with respect to the normal vector of the pixel p belonging to the subject area is calculated as α0−γp obtained by subtracting the angle γp formed by the representative normal vector and the normal vector of the pixel p from the angle α0, and this αp Is applied as the angle α in Equation 1 of the embodiment. By doing in this way, according to the direction of the surface which comprises a to-be-photographed object, the light quantity of the virtual light source which irradiates the surface can be calculated | required, and more highly accurate correction | amendment is attained. In order to reduce the amount of calculation, the normal vector may be determined not for each pixel but for each continuous area including a group of pixels within the subject area and having a luminance difference within a predetermined range.

  Hereinafter, a digital camera according to the second embodiment of the present invention will be described with reference to FIGS. In the first embodiment, a case has been described in which a target shadow state that is a target of relighting correction is set for each subject using a result of comparison of the shadow states of the subject for a plurality of subjects. In the second embodiment, the result of comparing the shadow states of the subjects is used to determine whether to set a target shadow state for each subject or to set a common target shadow state for a plurality of subjects. A method will be described. The overall configuration of the digital camera, the configuration of the image processing unit, and the overall relighting process in the second embodiment are the same as those in the first embodiment shown in FIGS. Therefore, explanation is omitted.

  In the second embodiment, the method for determining the target shadow state in step S305 in FIG. 3 is different from that in the first embodiment. With reference to FIG. 8, the determination method of the target shadow state in the second embodiment will be described in detail. The procedure in FIG. 8 is executed by the target shadow state setting unit 205.

In step S801, the target shadow state setting unit 205 compares the contrast value of the subject selected as the relighting target, and calculates the difference between the maximum value and the minimum value.
In step S802, the target shadow state setting unit 205 compares the calculated difference value with a predetermined contrast difference threshold value, and determines whether the difference value is smaller than the contrast difference threshold value. If it is determined that the difference value is smaller than the contrast difference threshold, the process proceeds to step S803. If it is determined that the difference value is greater than or equal to the contrast difference threshold, the process proceeds to step S806.

In step S803, the target shadow state setting unit 205 compares the brightness of the subject selected as the relighting correction target, and calculates the difference between the maximum value and the minimum value.
In step S804, the target shadow state setting unit 205 compares the calculated difference value with a predetermined brightness difference threshold value, and determines whether the difference value is smaller than the brightness difference threshold value. If it is determined that the difference value is smaller than the brightness difference threshold, the process proceeds to step S805. If it is determined that the difference value is equal to or greater than the brightness difference threshold, the process proceeds to step S806.

  The case where the process proceeds to step S805 corresponds to a case where the shadow of the subject caused by the ambient light is close to the subject. In this case, in step S805, the target shadow state setting unit 205 sets a common target shadow state for a plurality of subjects. As the common target shadow state, it is preferable to use a shadow state that is preferable in appearance.

  The case where the process proceeds to step S806 corresponds to a case where the manner in which the ambient light strikes varies greatly depending on the subject. In this case, if a common target shadow state is set for multiple subjects and relighting correction is performed, a subject whose shadow state changes greatly from the way the ambient light strikes during shooting will result in an unnatural impression. May be an image. For this reason, in step S806, the target shadow state setting unit 205 sets a target shadow state for each subject for a plurality of subjects. As a target value for each subject, a plurality of shade states that are preferable for appearance under a typical light source or photographing environment may be determined in advance and used. At that time, a plurality of shade states determined in advance are selected so as to have the closest shade state of the subject.

  The content of the processing for determining the setting value of the virtual light source after determining the target shadow state for each subject by the above processing and the processing for adding the image signal corresponding to the virtual light source are the same as in the case of the first embodiment. Therefore, detailed description is omitted.

  As described above, in this embodiment, when relighting correction is performed on a plurality of subject areas, a target shadow state is determined for each subject using the result of comparing the shaded state of the subject, or between subjects. Explained how to select whether to define a common target shadow state. As a result, when a plurality of subjects are corrected to a common shadow state and an unnatural image is obtained, a target shadow state is determined for each subject, and a natural relighting result can be obtained. Conversely, if the target shadow state is determined for each subject, the unnaturalness can be eliminated by defining a common target shadow state for an image in which the variation of the shadow state for each subject increases and becomes unnatural. .

[Modification 1]
In this embodiment, the case where it is determined whether or not the target shadow state is made common among the subjects based on the difference between the contrast and the brightness is described. However, the result of comparing the shadow states of a plurality of subjects. Any method may be used as long as it is a method for controlling the method for determining the target shadow state based on the above. For example, the determination may be made using only one of the brightness and contrast of the subject. Further, for example, in addition to the brightness and contrast of the subject, the shadow direction and subject color information are also used to determine whether the subject is exposed to the same ambient light or different ambient light. May be. In this case, if it is determined that the same ambient light is shining, a common target shadow state is set between the subjects, and if it is determined that different ambient light is shining, the ambient light shining on each of the subjects is set. A suitable target shadow state is set for each subject. By doing so, it becomes possible to perform relighting correction so as to be more visually appealing while retaining the characteristics due to the original ambient light.

[Modification 2]
Further, a method may be used in which the shade states of a plurality of subjects are compared and it is determined whether or not there is a subject with a shaded state that is visually favorable among them. In this case, when there is no subject that is in a shaded state that is visually appealing, the target shaded state is set to a shaded state that is common among the subjects. As the target shade state, a predetermined shade state that is desirable to the eye may be determined.

[Modification 3]
Further, in this embodiment, the case where a predetermined shadow state determined in advance is used as the target shadow state has been described, but the method for determining the target shadow state is not limited to this. For example, it is possible to detect the state of shading of a plurality of subjects with ambient light and determine the average value as the target shading state. In addition, when there are a large number of subjects in the same shadow state among a plurality of subjects, the shadow state may be selected as the target shadow state.

[Modification 4]
In addition, after setting the common target shadow state, the target shadow state is adjusted for each subject so that the relationship of the shadow state of the subject is not reversed by using the method described in the first embodiment. May be. By doing so, as in the case of the first embodiment, the shaded state does not reverse before and after the relighting correction, and a natural-looking relighting result can be obtained.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

50 System control unit, 101 Optical system, 102 Shutter, 103 Imaging unit, 104 A / D converter, 105 Image processing unit, 106 Image memory, 107 Memory control unit, 109 Display unit, 110 Codec unit, 111 Recording I / F 112 recording medium 113 face detection unit 120 operation unit 121 non-volatile memory 122 system memory

Claims (14)

Subject detection means for detecting a subject in the image of interest;
State detecting means for detecting a shadow state indicating a degree of shadow for each subject;
Based on the detected shadow state, a target for each target subject in the image of interest is arranged so that the order of the degree of shadow before correction of the subject and the order of shadow after correction are not interchanged. Target setting means for setting a target shadow state that is a shadow state to become,
A light source setting unit that determines a setting of a virtual light source for each target subject based on the shadow state detected by the state detection unit and the target shadow state for the target image;
An image processing apparatus comprising: a correction unit that corrects the brightness of the subject subject in the image of interest using the virtual light source.   When a plurality of subjects are detected by the subject detection unit, a subject selected from the plurality of subjects based on the shadow state of each subject detected by the state detection unit becomes the target. The image processing apparatus according to claim 1, wherein the image processing apparatus is a subject.   The shadow state is information representing a characteristic of a shadow generated in a subject area corresponding to the subject by ambient light illuminating the subject when the target image is captured. The image processing apparatus according to 1 or 2.   The shadow state includes information on brightness distribution in the subject area, information indicating contrast in the subject area, characteristics of the contour of the shadow area in the subject area, and the subject The image processing apparatus according to any one of claims 1 to 3, further comprising at least one piece of information representing an area of a shaded area.   In the target image, the target setting unit reduces the shadow level for a subject with the highest shadow level and increases the shadow level for a subject with the lowest shadow level in the target image. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is determined.   The target setting means calculates an evaluation value representing a shadow state of a subject for each of the plurality of subjects, and determines the target shadow state so that the evaluation value becomes a target evaluation value. The image processing apparatus according to any one of claims 1 to 5.   The image processing apparatus according to claim 6, wherein the evaluation value is a contrast value indicating a luminance ratio between a highest luminance portion and a lowest luminance portion in the subject area.   The target setting means sets the target shadow state to a different state for each subject or sets the same state between subjects based on a result of comparing the evaluation values for the plurality of subjects in the target image. The image processing apparatus according to claim 6, wherein the image processing apparatus determines whether or not.   The target setting means determines that the target shadow state is different for each subject when a difference in the evaluation values among the plurality of subjects is larger than a predetermined value. The image processing apparatus according to 8.   The image processing according to claim 6, wherein the target setting unit determines the target subject based on a result of comparing the evaluation values for the plurality of subjects. apparatus.   11. The setting of the virtual light source includes a position of the virtual light source with respect to the subject and an intensity of virtual illumination light emitted from the virtual light source. Image processing apparatus. The shaded state is indicated by a contrast value that is information representing contrast in the subject area,
When the target contrast value indicating the target shadow state is smaller than the contrast value indicating the shadow state detected by the state detection unit, the light source setting unit determines the position of the virtual light source as the normal of the subject. If the contrast value indicating the shadow state detected by the state detection means is smaller than the target contrast value indicating the target shadow state, the opposite side to the ambient light illuminating the subject, The image processing apparatus according to claim 11, wherein the position of the virtual light source is set on the same side as the ambient light with respect to the normal line of the subject. Subject detection means for detecting a subject in the image of interest;
State detecting means for detecting a shadow state indicating a degree of shadow for each subject;
Based on the detected shadow state, a target for each target subject in the image of interest is arranged so that the order of the degree of shadow before correction of the subject and the order of shadow after correction are not interchanged. Target setting means for setting a target shadow state that is a shadow state to become,
A light source setting unit that determines a setting of a virtual light source for each target subject based on the shadow state detected by the state detection unit and the target shadow state for the target image;
A program for causing a computer to function as correction means for correcting the brightness of the subject subject in the image of interest using the virtual light source. A subject detection step of detecting a subject in the image of interest;
A state detection step of detecting a shadow state indicating a degree of shadow for each subject;
Based on the detected shadow state, a target for each target subject in the image of interest is arranged so that the order of the degree of shadow before correction of the subject and the order of shadow after correction are not interchanged. A target setting step for setting a target shadow state, which is a shadow state to become,
A light source setting step for determining a setting of a virtual light source for each target subject based on the shadow state detected by the state detection step and the target shadow state for the target image;
And a correction step of correcting the brightness of the subject subject in the image of interest using the virtual light source.

Images