JP2020005084A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To make it easy for a user to set a virtual light source that can obtain a desired irradiation result.SOLUTION: An image processing apparatus (100) includes acquisition means (103) that acquires a captured image of a subject, setting means (50) that sets a virtual light source, correction means (114) that corrects the brightness of a specific subject in the captured image using the virtual light source, generating means (50) that generates a virtual light source state image representing a positional relationship between the subject and the virtual light source, and control means (50, 109) that superimposes the virtual light source state image on the captured image or the image after the brightness correction and displays the image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for correcting brightness or the like of an image.

  2. Description of the Related Art Conventionally, there has been known a lighting processing technique of irradiating a subject in a captured image with light from a virtual light source. As a result, it is possible to brighten a dark area such as a shadow caused by environmental light and obtain a preferable image. For example, Patent Literature 1 discloses, as a pseudo lighting processing technique for a captured image, a technique in which a brightness area lower than the average brightness of the entire face area is extracted as a shadow area, and the brightness of the extracted shadow area is increased. Have been. Thereby, the shadow of the face can be suppressed. In the technique of Patent Document 1, it is also possible to adjust the irradiation range, direction, and intensity of the virtual light source with respect to the subject by setting lighting parameters such as a writing area, a lighting direction, and a lighting intensity. Further, Patent Document 1 discloses that a parameter image representing a lighting parameter is superimposed and displayed on an image whose brightness has been corrected by the lighting process. The user can adjust the irradiation range, direction, and intensity of the virtual light source with respect to the subject by setting the lighting parameters while referring to the image after correction by the lighting process and the parameter image superimposed and displayed.

JP 2010-135996 A

  When correcting shadows by the lighting process, it is necessary to appropriately set the lighting parameters. If the lighting parameters are not set properly, the image after the lighting process becomes unnatural depending on a picture. . For example, when the virtual light source is irradiated on the subject from an obliquely upward direction, and when the virtual light source is irradiated on the subject from right beside, the degree of the light from the virtual light source on the subject is greatly different. For example, in the case of the method disclosed in Patent Document 1, the range, direction, and intensity of the virtual light source are expressed two-dimensionally in the parameter image. In that case, it is very difficult to set appropriate lighting parameters. In other words, the user refers to the two-dimensionally expressed parameter image, and supposes that the virtual light source is illuminated from an obliquely upward direction on the subject, which is also two-dimensionally expressed. A difficult setting operation such as setting the intensity lighting parameter is required. And even if such a difficult setting is performed, the irradiation result intended by the user is often not obtained.

  Therefore, an object of the present invention is to make it easy for a user to set a virtual light source that can obtain a desired irradiation result.

  An image processing apparatus according to the present invention includes: an obtaining unit that obtains a captured image of a subject; a setting unit that sets a virtual light source; and a correction that corrects the brightness of a specific subject in the captured image using the virtual light source. Means, generating means for generating a virtual light source state image representing a positional relationship between the specific subject and the virtual light source, and superimposing the virtual light source state image on the captured image or the image after the brightness correction. And control means for displaying the information.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to make it easy for a user to set the virtual light source which obtains a desired irradiation result.

FIG. 14 is a block diagram illustrating a configuration of an application example of the image processing apparatus. FIG. 3 is a block diagram illustrating a configuration of an image processing unit. FIG. 3 is a block diagram illustrating a configuration of a rewriting processing unit. It is a figure used for description of irradiation of a virtual light source, and its reflection. It is a figure showing an example of an image before and after rewriting processing. 5 is a flowchart of a rewriting process according to the first embodiment. It is a figure explaining a rewriting recommendation mark. FIG. 7 is a diagram illustrating an example of a virtual light source state image. It is a flowchart of the rewriting process according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations described in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.
<First embodiment>
Hereinafter, a digital camera will be described in a first embodiment as an application example of the image processing apparatus according to the present invention.
FIG. 1A is a block diagram illustrating a schematic configuration example of a digital camera 100 according to the present embodiment. The configuration of FIG. 1B will be described in a second embodiment described later.

The lens group 101 is an imaging optical system including a focus lens, a zoom lens, and a diaphragm, and forms an optical image of a subject or the like on an imaging element included in the imaging unit 103. The shutter 102 is, for example, a focal plane shutter, and opens and closes under the control of the system control unit 50.
The imaging unit 103 converts an optical image formed on the imaging element via the lens group 101 into an electric signal by photoelectric conversion and outputs an analog image signal. The imaging device of the imaging unit 103 is a CCD image sensor or a CMOS image sensor. On the imaging surface of the imaging device, color filters corresponding to the three primary color components of R (red), G (green), and B (blue) are provided for each pixel in a so-called Bayer array. For this reason, the imaging unit 103 outputs an analog image signal composed of color signals of RGB pixels corresponding to the Bayer array.
The A / D conversion unit 104 converts the analog image signal output from the imaging unit 103 into digital image data, and outputs the image data to the image processing unit 105 and the memory control unit 107.

  The distance measurement sensor 123 is a sensor that measures a subject distance from the digital camera 100 to the subject, and outputs distance information corresponding to a pixel unit of an image captured by the imaging unit 103. The pixel-by-pixel distance information output from the distance measurement sensor 123 is temporarily stored in the image memory 106 via the memory control unit 107 and sent to the image processing unit 105 described later.

The image memory 106 stores image data captured by the image capturing unit 103 and digitally converted by the A / D converter 104 under the control of the memory control unit 107, and distance information in pixel units from the distance measurement sensor 123. Remember temporarily. The image memory 106 also stores image data read from the recording medium 112 via the I / F unit 111, image data to be displayed on the display unit 109, and the like.
The recording medium 112 is, for example, a semiconductor memory card, a card-type hard disk, or the like, and is detachable from the digital camera 100. Image data taken by the digital camera 100 or image data taken by another digital camera Etc. are recorded.
The I / F unit 111 is an interface unit that electrically connects the recording medium 112 and the digital camera 100. Image data photographed by the digital camera 100 is sent to and recorded on the recording medium 112 via the I / F unit 111, and image data read from the recording medium 112 is transmitted via the I / F unit 111. The data is sent to the image memory 106 and the like.

  The face detection unit 113 performs a predetermined image analysis process on an image captured by the imaging unit 103, an image read from the image memory 106, an image read from the recording medium 112, and the like. An area where a human face is reflected is detected from the image. The data of the face detection result detected by the face detection unit 113 is temporarily stored in the image memory 106, and is read out from the image memory 106 as needed, and is sent to the image processing unit 105 and the system control unit 50 to be described later. Sent.

The D / A converter 108 captures image data from the imaging unit 103 and outputs the image data from the A / D conversion unit 104, the image data read from the image memory 106, and the image data read from the recording medium 112. Are converted into analog video signals and output to the display unit 109.
The display unit 109 has a display device such as an LCD (liquid crystal display), and displays an image based on the analog video signal output from the D / A converter 108. The display unit 109 displays a captured image, a live view image captured by the image capturing unit 103 during so-called live view display, and the like. The display unit 109 also displays a user interface image when the user operates the digital camera 100 or the like.
The operation unit 120 includes a shutter button and various switches of the digital camera 100, a touch panel disposed on a screen of the display unit 109, and the like, detects an operation by a user, and notifies the system control unit 50 of the detected operation information. I do.

  The codec unit 110 performs compression encoding / decoding of image data. The codec 110 compresses and encodes the image data stored in the image memory 106 in a format conforming to a standard such as JPEG or MPEG. The image data compressed and encoded by the codec unit 110 is recorded on the recording medium 112 via the I / F unit 111 under the control of the system control unit 50. Also, the codec unit 110 can decode the compression-encoded image data read from the recording medium 112 via the I / F unit 111 under the control of the system control unit 50. .

The nonvolatile memory 121 includes, as an auxiliary storage device, a nonvolatile semiconductor memory such as an EEPROM that stores programs, parameters, and the like.
The system memory 122, as a main storage device, stores programs and the like read from the nonvolatile memory 121, and stores constants and variables for operation of the system control unit 50. That is, in the system memory 122, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are expanded.

  The system control unit 50 includes a CPU or an MPU, executes a program stored in the nonvolatile memory 121 by expanding the program in a work area of the system memory 122, and executes each function of the entire digital camera 100 of the present embodiment. Control. For example, the system control unit 50 controls various operations such as control of shooting, control of various image processes, control of a rewriting process described later, switching of a camera operation mode, and the like, based on an operation input and a setting state of the operation unit 120 by the user. Perform control. The program according to the present embodiment includes, in addition to the basic operation program at the time of photographing a subject in the digital camera 100, a program for controlling rewriting processing to be described later, superimposing a virtual light source state image, and the like. The program executed by the system control unit 50 may be obtained not only from the case where it is stored in the nonvolatile memory 121 in advance, but also from the recording medium 112 or the like, or may be obtained via a network (not shown). .

  The image processing unit 105 performs various kinds of processing such as white balance processing, γ (gamma) processing, contour enhancement, and color correction processing on the image captured by the imaging unit 103 or the image read from the image memory 106. Perform image processing. The image processing unit 105 also outputs information on a face detection result by the face detection unit 113, captured image data, a two-dimensional distance map generated from pixel-by-pixel distance information detected by the distance measurement sensor 123, and the like. Then, an evaluation value used for exposure control, distance measurement control, and the like is calculated. Information on the calculated evaluation value is sent to the system control unit 50.

  At this time, based on the evaluation value calculated by the image processing unit 105, the system control unit 50 performs, for example, exposure control for adjusting exposure to the face of the subject, distance measurement control for adjusting focus to the face, and the like. Specifically, the system control unit 50 controls so-called TTL (through-the-lens) type AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and the like. Various known techniques can be used for the evaluation value used for performing the exposure control, the distance measurement control, and the like, and the calculation processing thereof, and the description and illustration thereof are omitted here.

  The rewriting processing unit 114 performs a rewriting process (writing process) on the image captured by the imaging unit 103 and the image read from the image memory 106. The relighting process according to the present embodiment is a process of correcting the brightness by irradiating light from a virtual light source to an area such as a specific subject in an image. The details of the rewriting process according to the present embodiment will be described later.

<Schematic configuration of image processing unit>
FIG. 2 is a block diagram illustrating a main configuration of the image processing unit 105.
As shown in FIG. 2, the image processing unit 105 includes a synchronization processing unit 200, a WB amplification unit 201, a luminance / color generation unit 202, a contour emphasis unit 203, a luminance gamma processing unit 204, a color conversion unit 205, and a color gamma processing. Unit 206, a color difference generation unit 207, a shadow acquisition unit 208, and the like.

  In the image processing unit 105, for example, image data captured by the imaging unit 103 and digitally converted by the A / D conversion unit 104 is input to the synchronization processing unit 200. The image data input to the synchronization processing unit 200 is R, G, and B data corresponding to the Bayer arrangement as described above. The synchronization processing unit 200 performs synchronization processing on image data corresponding to the Bayer array, and generates R, G, and B color signals (hereinafter, referred to as RGB signals). The RGB signals generated by the synchronization processing section 200 are sent to the WB amplification section 201.

  The WB amplifying unit 201 performs a process of adjusting the white balance by applying a gain to the RGB signal based on the white balance gain value calculated by the system control unit 50. The RGB signals whose white balance has been adjusted by the WB amplifying unit 201 are sent to the luminance / color generating unit 202.

  The luminance / color generation unit 202 generates a luminance signal Y from the RGB signals after white balance adjustment, and outputs the generated luminance signal Y to the contour emphasis unit 203. Further, the luminance / color generation unit 202 outputs the RGB signals to the color conversion unit 205.

  The contour emphasizing unit 203 performs a contour emphasizing process on the image composed of the luminance signal Y supplied from the luminance / color generating unit 202, and outputs the luminance signal Y after the contour emphasizing process to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y supplied from the contour emphasizing unit 203. The luminance signal Y after the gamma correction is sent to the image memory 106 and is temporarily stored.

  The color conversion unit 205 generates a RGB signal having a desired color balance by performing a matrix operation or the like on the RGB signals supplied from the luminance / color generation unit 202, and outputs a color gamma processing unit 206 and a shadow acquisition unit. 208. The color gamma processing unit 206 performs gamma correction on the RGB signals supplied from the color conversion unit 205, and outputs the result to the color difference generation unit 207. The color difference generation unit 207 generates color difference signals RY and BY from the RGB signals after the color gamma processing by the color gamma processing unit 206. The color difference signals RY and BY are sent to the image memory 106 and are temporarily stored.

  The image data composed of the luminance signal Y and the chrominance signals RY and BY temporarily stored in the image memory 106 is then transferred to the codec of FIG. Sent to the unit 110. As a result, the codec 110 compresses and encodes the image data including the luminance signal Y and the color difference signals RY and BY based on a predetermined standard. The data compressed and encoded by the codec unit 110 is recorded on the recording medium 112 via the I / F unit 111 under the control of the system control unit 50 as described above.

  The shadow obtaining unit 208 obtains information for analyzing the state of the shadow generated on the subject by the environmental light source. Further, in addition to the RGB signals from the color conversion unit 205, when a face is detected by the above-described face detection unit 113, information on the face area is also input to the shadow acquisition unit 208. Then, the shadow obtaining unit 208 obtains average brightness information of the subject, brightness histogram information of the face area, and the like as shadow information. The shadow information obtained by the shadow obtaining unit 208 is sent to the system control unit 50.

<Schematic configuration of rewriting processing unit>
FIG. 3 is a block diagram illustrating a schematic configuration of the relighting processing unit 114.
In the digital camera 100 of the present embodiment, the rewriting process is performed when the camera operation mode for executing the rewriting process is selected by the user. When the camera operation mode for executing the rewriting process is selected, the luminance signal Y and the color difference signals RY and BY read from the image memory 106 are input to the rewriting processing unit 114.

  3, the relighting processing unit 114 includes an RGB conversion unit 301, a degamma processing unit 302, a reflection component calculation unit 305, an addition processing unit 306, a gamma processing unit 307, and a luminance / color difference conversion unit 308. Although the distance calculation unit 303 and the normal line calculation unit 304 shown in FIG. 3 are provided, for example, in the image processing unit 105, they may be included in the rewriting processing unit 114. , May be included in the system control unit 50.

The RGB conversion unit 301 converts the input luminance signal Y and color difference signals BY and RY into RGB signals, and outputs the converted RGB signals to the degamma processing unit 302.
The degamma processing unit 302 performs an arithmetic process (that is, degamma processing) on the RGB signal input from the RGB conversion unit 301 using the inverse characteristic of the gamma characteristic used in the color gamma processing unit 206 of the image processing unit 105. By doing so, the data is converted into RGB linear data. Then, the degamma processing unit 302 outputs the RGB signals (Rt signal, Gt signal, Bt signal) after the degamma processing to the reflection component calculation unit 305 and the addition processing unit 306.

The distance calculation unit 303 calculates a two-dimensional distance map from the distance information acquired for each pixel of the captured image by the distance measurement sensor 123.
The normal calculating unit 304 calculates a normal map of an area such as a face detected by the face detecting unit 113 based on the distance map calculated by the distance calculating unit 303. The process of generating a normal map from a distance map will be described with reference to FIG.

  FIG. 4 is a diagram showing a positional relationship between the shooting coordinates of the digital camera 100 and a specific subject 401 such as a face. By using the two-dimensional distance map based on the above-described pixel-based distance information of the captured image, for example, as shown in FIG. 4, for the subject 401, the difference of the distance D in the depth direction with respect to the horizontal difference ΔH of the captured image ΔD can be calculated. Further, the gradient can be calculated from the difference ΔH in the horizontal direction and the difference ΔD of the distance D in the depth direction, and the normal N can be calculated from the gradient. The normal line calculation unit 304 calculates the normal line N corresponding to each pixel of the captured image by performing such processing on the captured image. Then, the normal calculation unit 304 generates a two-dimensional normal map based on each normal N corresponding to each pixel of the captured image and outputs the map to the reflection component calculation unit 305.

  The reflection component calculation unit 305 calculates a component of the virtual light source 402 reflected by the subject 401 based on the distance K from the virtual light source 402 to the subject 401, the normal N, and the direction vector L of the virtual light source 402. calculate. For example, the reflection component calculation unit 305 performs shooting in such a manner that the reflection component calculation unit 305 is inversely proportional to the square of the distance K from the virtual light source 402 to the subject 401, and is proportional to the inner product of the vector of the normal line N of the subject 401 and the direction vector L of the virtual light source 402. The reflection component at the coordinate position corresponding to each pixel of the image is calculated. In the example of FIG. 4, the reflection component at the horizontal pixel position H1 of the captured image is proportional to the inner product of the normal N1 at the coordinates of the horizontal pixel position H1 and the direction vector L1 of the virtual light source 402. This value is inversely proportional to the square of the distance K1. For the sake of simplicity, description of the reflection component at the vertical pixel position will be omitted.

  Here, the reflection components (Ra, Ga, Ba) of the virtual light source from the subject reflected by the subject can be represented by the following equation (1). Note that α in Expression (1) is the intensity of the virtual light source and is a gain value of the rewriting correction amount. In the equation, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, K is the distance from the virtual light source to the subject, and Rt, Gt, and Bt are the RGB signals after the degamma process described above. .

Ra = α × (−L · N) / K ＾ 2 × Rt
Ga = α × (−L · N) / K ＾ 2 × Gt Equation (1)
Ba = α × (−L · N) / K ＾ 2 × Bt

The reflection components (Ra, Ga, Ba) calculated by the reflection component calculation unit 305 as described above are output to the additional processing unit 306.
The addition processing unit 306 performs a process of adding a virtual light source component (Ra, Ga, Ba) to the RGB signals (Rt, Gt, Bt) after the degamma processing. The addition processing unit 306 generates the RGB signals (Rout, Gout, Bout) after the addition processing by performing the virtual light source addition processing represented by the following equation (2).

Rout = Rt + Ra
Gout = Gt + Ga Equation (2)
Bout = Bt + Ba

The RGB signals (Rout, Gout, Bout) output from the additional processing unit 306 are input to the gamma processing unit 307.
The gamma processing unit 307 performs gamma correction on the RGB signals (Rout, Gout, Bout). The gamma processing unit 307 performs gamma correction based on the gamma characteristics used in the color gamma processing unit 206 of the image processing unit 105, and outputs the RGB signals (R'out, G'out, B'out) after the gamma correction. Output to the luminance / color difference conversion unit 308.

  The luminance / color difference conversion unit 308 generates a luminance signal Y and color difference signals RY, BY from the input RGB signals (R'out, G'out, B'out).

  The above is the operation of the rewriting processing unit 114. FIGS. 5A and 5B are diagrams showing examples of images before and after the rewriting process. FIG. 5A shows an example of the captured image 500 before the relighting process, and FIG. 5B shows an example of the image 501 after the relighting process. The relighting process is a process in which light from the virtual light source 520 is virtually applied to the subject 510 that is dark in the captured image 500 before the process as illustrated in FIG. For this reason, after the rewriting process, as shown in FIG. 5B, a brightly corrected image 501 of the subject 511 is obtained.

  In the present embodiment, the system control unit 50 stores the luminance signal Y and the color difference signals RY and BY output from the relighting processing unit 114 in the image memory 106 via the memory control unit 107, and then stores the codec unit At 110, compression encoding is performed. Further, the system control unit 50 controls the recording unit 112 to record the image on the recording medium 112 via the I / F unit 111 and display the image after the rewriting process on the display unit 109.

  The above-described image processing unit 105 in FIG. 2 and the relighting processing unit 114 in FIG. 3 have not only a hardware configuration but also a software configuration that realizes processing of each unit in FIGS. 2 and 3 by executing a program by a CPU or the like. May be realized by: Further, a part may be realized by a hardware configuration and the rest may be realized by a software configuration. The program for this software configuration is not limited to the case where it is prepared in advance, but may be obtained from a recording medium such as an external memory (not shown), or may be obtained via a network (not shown).

<Superimposed display of virtual light source state image>
When displaying the image after the relighting process on the display unit 109, the digital camera 100 according to the present embodiment converts the virtual light source state image representing the positional relationship between the virtual light source and the subject into a captured image or an image after the relighting process. It also has the function of superimposed display.
FIG. 6 shows a flowchart of a process from execution of the relighting process on the captured image to superimposed display of the virtual light source state image. The process of the flowchart illustrated in FIG. 6 is realized by the system control unit 50 executing the program according to the present embodiment and controlling each unit of the digital camera 100. In the following description, steps S601 to S616 of each process are abbreviated as S601 to S616. The same applies to other flowcharts described later.

  In the flowchart shown in FIG. 6, the processes from S601 to S606 are processes executed before the main photographing is performed in the digital camera 100. Note that the shutter button of the operation unit 120 of the digital camera 100 of the present embodiment is a button that can take two states, a half-pressed state and a fully-pressed state. When the shutter button is half-pressed, the system control unit 50 controls the digital camera 100 to an operation state before the main shooting in which AF processing, AE processing, AWB processing, face detection processing, and the like are performed. Then, when the shutter button is half-pressed, the system control unit 50 controls each unit so that the digital camera 100 performs the main shooting. The camera operation when the shutter button is half-pressed or full-pressed is the same as that of a general camera, and a detailed description thereof will be omitted.

In step S601, when the system control unit 50 detects from the operation information of the operation unit 120 that the user has half-pressed the shutter button, the process proceeds to step S602.
In S602, the system control unit 50 determines whether or not the camera operation mode is set to a mode for executing the rewriting process, that is, whether or not to execute the rewriting process. For example, when the camera operation mode in which the user operates the operation unit 120 to execute the rewriting process is set in advance, the system control unit 50 advances the process to S603. On the other hand, when the operation mode for executing the relighting process is not set as the camera operation mode, the system control unit 50 ends the processing of the flowchart in FIG.

  In step S603, if the face to be shot is detected by the face detection unit 113 from the image obtained before the main shooting, the system control unit 50 obtains the face detection result. Specifically, information such as the coordinate position of the face and the reliability of the coordinate position is obtained from the face detection unit 113.

  Next, in steps S604 and S605, the system control unit 50 controls the image processing unit 105 to execute the shadow information acquisition processing in the shadow acquisition unit 208. At this time, in step S604, the shadow acquisition unit 208 determines, as luminance information in the face area according to the result of the face detection by the face detection unit 113, a luminance distribution such as an average luminance value in the face area or a luminance histogram in the face area. Get information such as. In step S605, the shading obtaining unit 208 obtains information such as an average brightness value and a brightness distribution such as a brightness histogram outside the face area as brightness information outside the face area. After S605, the system control unit 50 proceeds to S606.

  In step S606, the system control unit 50 determines from the image obtained before the main shooting whether the shooting target scene (scene) is a rewriting target scene that is effective when the rewriting process is performed. I do. In the case of the present embodiment, the system control unit 50 determines whether or not the scene is a rewriting target scene based on the luminance information in the face area and the luminance information outside the face area acquired by the shading acquisition unit 208 in S604 and S605. I do.

  For example, in a scene in which the average luminance value in the face area acquired in step S604 is extremely low and there are many portions that are crushed black, or conversely, in a scene where the average luminance value is extremely high and there are many portions that are overexposed, the rewriting process is performed. It is considered that even if the method is applied, a very effective result cannot be obtained. For this reason, in the case of a scene where the average luminance value in the face region is extremely low and is crushed by black, or conversely, if the scene is extremely high and there are many portions that are overexposed, the system control unit 50 determines in S606 that the rewriting target It is determined that the scene is not a scene. For example, in a scene in which the luminance distribution of the luminance histogram in the face region is deviated, it is considered that a relighting effect can be obtained by irradiating the virtual light source to the deviated region to reduce the luminance deviation. For this reason, in the case of a scene in which the luminance distribution of the luminance histogram in the face area is uneven, the system control unit 50 determines in S606 that the scene is a rewriting target scene. For example, in a scene in which the luminance distribution of the luminance histogram is flat and there is no sense of unevenness of the face, the virtual light source reflection components (Ra, Ga, Ba) are subtracted from the RGB signals (Rt, Gt, Bt) to reduce the flatness of the luminance distribution. It is thought that the effect of relighting can be obtained if it is reduced. Therefore, in the case where the luminance distribution of the luminance histogram in the face area is flat, the system control unit 50 determines in S606 that the scene is a rewriting target scene. Further, for example, in a backlight scene in which the average luminance value inside the face area is low and the average luminance value outside the face area is high, it is considered that the relighting effect appears. Therefore, in the case of a backlight scene where the average luminance value in the face area is low and the average luminance value outside the face area is high, the system control unit 50 determines that the scene is a rewriting target scene in S606.

  Further, the system control unit 50 can also superimpose a mark, a message, or the like according to the determination result of the rewriting target scene in S606 on the image displayed on the display unit 109. For example, when it is determined that the scene is a rewriting target scene, a recommendation mark or the like indicating that the scene is a recommended rewriting scene is displayed in a superimposed manner to notify the user that the rewriting process is effective. It becomes possible.

  FIG. 7 is a diagram illustrating an example in which a recommendation mark 701 indicating a rewriting recommended scene is superimposed on an image 700 displayed on the display unit 109 when it is determined that the scene is a rewriting target scene. . In the example of FIG. 7, a recommendation mark 701 indicating a rewriting recommended scene is superimposed on, for example, the upper left of the image 700 displayed on the display unit 109. This makes it possible to inform the user that the rewriting process is effective. Further, the system control unit 50 may display a rewriting process mode ON / $ FF switch button 702 near the recommendation mark 701 of the image 700. Then, when an instruction to turn on the switch button 702 is given via the touch panel of the operation unit 120, the system control unit 50 switches the camera operation mode to a mode in which the rewriting process is performed. In the case of displaying the ON / $ FF switch button 702 of the rewriting process mode, the system control unit 50 proceeds to S603 without performing the determination process of S602 after S601 of FIG. .

When the system control unit 50 determines in S606 that the scene is a rewriting target scene as described above, the process proceeds to S607. On the other hand, if it is determined that the scene is not a rewriting target scene, the system control unit 50 ends the processing of the flowchart in FIG.
If the process has proceeded to S607, the system control unit 50 controls each unit so that the digital camera 100 executes actual shooting when the user fully presses the shutter button of the operation unit 120.

  Next, in S608, the system control unit 50 controls the distance calculation unit 303 to calculate a two-dimensional distance map. At this time, the distance calculation unit 303 generates a weighted map image (mapK) based on the distance from the digital camera 100 to the subject or the like under the control of the system control unit 50. Specifically, the distance calculation unit 303 calculates the distance K in pixel units based on the pixel-based distance information of the captured image acquired from the distance measurement sensor 123, and further calculates 1 / K１ ／ 2 in pixel units. Calculate the value normalized by the bit width. Then, the distance calculation unit 303 generates a two-dimensional array of the normalized values corresponding to each pixel as a distance-weighted map image (mapK).

  Next, in step S609, the system control unit 50 controls the normal line calculation unit 304 to calculate a normal line map. At this time, under the control of the system control unit 50, the normal line calculation unit 304 generates a normal line map image (mapN) based on the weighted map image (mapK) based on the distance acquired from the distance calculation unit 303. Specifically, the normal vector calculation unit 304 calculates the normal vector (N) of the subject 401 in units of pixels as described with reference to FIG. 4, obtains the direction cosine for each of these coordinate axis directions, and calculates the direction cosine. Expressed by an arbitrary bit width, a weight map (mapN) based on the normal is used.

Next, in S610, the system control unit 50 controls the reflection component calculation unit 305 of the relighting processing unit to calculate the reflection component of the virtual light source. At this time, the system control unit 50 sends to the reflection component calculation unit 305 the coordinate information of the face detected by the face detection unit 113 and the luminance information inside and outside the face area acquired by the shadow acquisition unit 208. Then, under the control of the system control unit 50, the reflection component calculation unit 305 determines the position, irradiation range, and intensity of the virtual light source based on the information. For example, when the luminance distribution in the face region is biased, the reflection component calculation unit 305 determines the position, the irradiation range, and the intensity of the virtual light source such that the virtual light source shines on a region having a low luminance value. Here, for example, assuming that the coordinates of an area having a low luminance value are (x1, y1), the reflection components (Ra _{(x1, y1)} , Ga _{(x1, y1)} , and Ba _{(x1, y1 )} reflected from the subject by the virtual light source. ₎ ) Is represented by the following equation (3).

Ra _{(x1, y1)} = α × (−L _{(x1, y1)} · N _{(x1, y1)} ) / K _{(x1, y1)} ＾ 2 × Rt
Ga _{(x1, y1)} = α × (−L _{(x1, y1)} · N _{(x1, y1)} ) / K _{(x1, y1)} ＾ 2 × Gt Equation (3)
Ba _{(x1, y1)} = α × (−L _{(x1, y1)} · N _{(x1, y1)} ) / K _{(x1, y1)} ＾ 2 × Bt

In the equation (3), Rt, Gt, and Bt are RGB signals after the above-described degamma processing, α is the intensity of the virtual light source, L _{(x1, y1)} is a three-dimensional direction vector of the virtual light source, and N _{(x1 , y1)} is a three-dimensional normal vector of the subject located at the position of (x1, y1) coordinates. Further, K _{(x1, y1)} in Expression (3) is the distance between the virtual light source and the subject existing at the position of (x1, y1) coordinates.

Further, under the control of the system control unit 50, the reflection component calculation unit 305 determines the irradiation range of the virtual light source, the position of the virtual light source, and the intensity of the virtual light source based on Expression (3). In the case of the present embodiment, the reflection component calculation unit 305 acquires information on each coordinate having a luminance value lower than an arbitrarily set threshold, based on the information on the luminance distribution in the face area acquired by the shadow acquisition unit 208. Then, the irradiation range of the virtual light source is determined so that those coordinates are included. In addition, the reflection component calculation unit 305 determines the intensity α of the virtual light source and the distance K ₍ from the virtual light source to the subject) so that the light from the virtual light source falls on a region (coordinates (x1, y1)) having a low luminance value. _{x1, y1)} . The position of the virtual light source is specified based on the three-dimensional direction vector L _{(x1, y1)} and the distance K _{(x1, y1)} of the virtual light source.

Here, in order for the virtual light source to hit an area (coordinates (x1, y1)) where the luminance value of the subject is low, the reflection components (Ra _{(x1, y1)} , Ga _{(x1, y1)} , Ba _{(x1, y1) It} is necessary to determine the intensity α of the virtual light source and the distance K _{(x1, y1)} such that the value of ₎ ) becomes a positive value. Also, for example, if the intensity α of the virtual light source is set too high, overexposure and gradation inversion may occur. Therefore, the intensity α of the virtual light source is set to a value within a threshold range obtained in advance. It is desirable. For this reason, the reflection component calculation unit 305 calculates, as a threshold range for the intensity of the virtual light source, a luminance value range of the average luminance value ± β of a region having a high luminance value outside the irradiation range of the virtual light source, and For example, the average value is determined as the intensity α of the virtual light source. The reflection component calculation unit 305 determines the range of the distance from the virtual light source to the subject such that the reflection components (Ra _{(x1, y1)} , Ga _{(x1, y1)} , and Ba _{(x1, y1)} ) have positive values. Is calculated, and an average value of the range of the distance is determined as the distance K _{(x1, y1)} . Further, the reflection component calculation unit 305 calculates a range including each position of the virtual light source such that the reflection components (Ra _{(x1, y1)} , Ga _{(x1, y1)} , and Ba _{(x1, y1)} ) have positive values. , The position range of the virtual light source.

  Next, in S611, the system control unit 50 controls the reflection component calculation unit 305 to generate a weighted map image using the virtual light source. At this time, the reflection component calculation unit 305 calculates the light source direction vector -L of the virtual light source in pixel units, obtains the direction cosine for each coordinate axis direction in pixel units, and expresses the direction cosine with an arbitrary bit width. A weighting map (mapL) based on the virtual light source is used.

  Next, in S612, the system control unit 50 controls the reflection component calculation unit 305 to calculate the reflection component of the virtual light source in the subject. Here, the reflection components (Ra, Ga, Ba) of the virtual light source can be calculated by using the aforementioned equation (1). The reflection component calculation unit 305 replaces the expression (1) with the distance-weighted map image (mapK), the weighted map image based on the normal of the subject (mapN), and the weighted map image (mapL) based on the virtual light source. Is used to calculate the reflection component of the virtual light source.

Ra = α × mapL ・ mapN ・ mapK × Rt
Ga = α × mapL · mapN · mapK × Gt Equation (4)
Ba = α × mapL ・ mapN ・ mapK × Bt

  Next, in S613, the system control unit 50 generates a virtual light source state image. Specifically, the system control unit 50 generates a virtual ray state image in which each parameter used by the reflection component calculation unit 305 to calculate the virtual light source reflection component is illustrated.

Hereinafter, an example of the virtual light source state image will be described with reference to FIGS. 8A to 8E.
FIG. 8A is a diagram illustrating an example in which a three-dimensional virtual light source state image 802 is superimposed and displayed at the upper right position of the image 800 after the rewriting process. Further, a mark 801 indicating that the rewriting process has been performed is displayed at, for example, the upper left position of the image 800. The virtual light source state image 802 includes a position of the subject 803, a position of the virtual light source 806, an irradiation range 808 of the virtual light source 806, a position of the camera 805, and a position of the virtual light source 806 in a three-dimensional space of the x-axis, the y-axis, and the z-axis. This is an image in which an intensity bar 804 indicating the intensity is arranged. The position of the virtual light source 806, the irradiation range 808, and the intensity bar 804 of the virtual light source state image 802 can be operated by a user via an operation unit 120 such as a touch panel provided on the display unit 109. When the position of the virtual light source 806, the irradiation range 808, and the intensity bar 804 of the virtual light source state image 802 are operated by the user via the operation unit 120, the system control unit 50 determines the position of the virtual light source according to the operation. , The irradiation range, the intensity of the virtual light source, etc. are reset.

  FIG. 8B shows another example of the virtual light source state image 810. As in the example of FIG. 8A, the position of the subject 813, the position of the virtual light source 816, the irradiation range 818 of the virtual light source 816, This is an image in which the position of the camera 815, the intensity bar 814, and the like are arranged. In the virtual light source state image 810 of FIG. 8B, an auxiliary line 819 serving as a guide when the user operates the position of the virtual light source and the irradiation range is further included in the x-axis, y-axis, and z-axis (of the z-axis). The auxiliary line is not shown.). In the case of the virtual light source state image 810 in FIG. 8B, since the auxiliary line 819 is provided, the user can easily operate the position and the irradiation range of the virtual light source.

  FIG. 8C shows another example of the virtual light source state image 820, which is an image represented by a plan view when the positional relationship between the subject 823 and the virtual light source 826 is viewed from directly above. The virtual light source state image 820 in FIG. 8C is an image in which the subject 823, the virtual light source 826, the irradiation range 828 of the virtual light source 826, the position of the camera 825, the intensity bar 824, and the like are arranged. In the virtual light source state image 820, the position of the virtual light source 826, the irradiation range 828, and the intensity bar 824 of the virtual light source state image 820 can be operated by the user via the operation unit 120, as described above. In the case of the virtual light source state image 820 of FIG. 8C, the user can easily grasp the two-dimensional positional relationship between the subject 823 and the virtual light source 826.

  FIGS. 8D and 8E are examples in which the range of each parameter calculated by the reflection component calculation unit 305 in step S610 is displayed as a guide line. FIG. 8D is an example in which a guideline 837 is displayed on a virtual light source state image 830 similar to the virtual light source state image 802 in FIG. The position of the subject 833, the position of the virtual light source 836, the irradiation range 838 of the virtual light source 836, the position of the camera 835, the intensity bar 834 indicating the intensity of the virtual light source 836, and the like in FIG. 8D are the same as those in FIG. It is similar. FIG. 8E is an example in which the guideline 847 is displayed on a virtual light source state image 840 similar to the virtual light source state image 820 in FIG. 8C. The position of the subject 843, the position of the virtual light source 846, the irradiation range 848 of the virtual light source 846, the position of the camera 845, the intensity bar 844 indicating the intensity of the virtual light source 846, and the like in FIG. 8E are the same as those in FIG. It is similar. In the case of FIG. 8D and FIG. 8E, the user operates the position of the virtual light source and the range of the virtual light source, for example, according to the guideline, thereby changing the parameters of the position of the virtual light source and the range of the virtual light source. It can be adjusted. By displaying the guidelines as shown in FIGS. 8D and 8E and restricting the user's operation to the operations in accordance with the guidelines, for example, the parameters of the virtual light source that may break the picture Setting can be avoided.

The description returns to the flowchart of FIG.
In S614, the system control unit 50 controls the rewriting processing unit 114 to execute the rewriting process. More specifically, the additional processing unit 306 outputs the reflection components (Ra, Ga, and Ba) of the virtual light source calculated by the reflection component calculation unit 305 to the output of the degamma processing unit 302 (Expression (2)). Rt, Gt, Bt) to perform rewriting processing.

Next, in step S615, the system control unit 50 causes the display unit 109 to display a display image in which the virtual light source state image generated in step S613 is superimposed on the image after the relighting process.
Thereafter, in S616, the system control unit 50 determines whether or not the user has performed a change operation on the virtual light source state image via the operation unit 120. If the system control unit 50 determines that an operation for changing the virtual light source state image has been performed, the process returns to S611, and performs the above-described control from S611. On the other hand, when it is determined that the operation of changing the virtual light source state image has not been performed, the processing of the flowchart in FIG. 6 ends.

  As described above, the digital camera 100 according to the first embodiment generates a virtual light source state image in which parameters such as the position, irradiation range, and intensity of the virtual light source are schematized, and superimposes the captured image or the image after the rewriting process on the captured image. It has the function to do. According to the digital camera 100 of the present embodiment, the user can easily adjust each parameter of the virtual light source by adjusting the position, irradiation range, and intensity of the virtual light source (light) based on the displayed virtual light source state image. , And a desired relighting processing result can be obtained.

  In the present embodiment, an example of a three-dimensional view or a plan view is given as an example of the virtual light source state image. However, if the virtual light source state image allows the user to grasp the positional relationship between the subject and the virtual light source, the above-described virtual light source state image can be used. It is not limited to the example. Further, in the present embodiment, the case where there is one virtual light source is described as an example, but the present invention is not limited to this. For example, a configuration may be employed in which relighting processing is performed by a plurality of light sources such that one virtual light source irradiates light from a diagonally upper right to the subject and another virtual light source irradiates light from right beside the subject. In this case, in the virtual light source state image, parameters such as position, irradiation range, and intensity can be adjusted for the plurality of virtual light sources in the same manner as described above.

<Second embodiment>
Next, as an application example of the image processing apparatus according to the present invention, in the second embodiment, an information processing apparatus such as a personal computer (hereinafter, referred to as a PC) will be described. The PC according to the second embodiment acquires a captured image and additional information such as distance information from a digital camera or the like. Then, the PC generates an image after the rewriting process on the captured image, generates a virtual light source state image, and superimposes and displays the virtual light source state image on the captured image or the image after the relighting process. Note that the additional information acquired together with the captured image may include a face detection result obtained by the digital camera and normal line information in addition to the distance information. If the additional information does not include the face detection result or the normal line information, the PC according to the second embodiment detects the face area from the captured image to generate the face detection result, and the normal line information from the distance information. Is calculated.

FIG. 1B is a diagram illustrating a schematic configuration example of a PC 150 according to the second embodiment.
The PC 150 according to the second embodiment includes a CPU 151, a ROM 152, a RAM 153, a display unit 154, an operation unit 156, a recording medium 157, an I / F unit 158, and the like.

  The CPU 151 performs overall control of the entire operation of the PC 150 and performs various arithmetic processes, and is connected to each component (152 to 159) via a system bus. The ROM 152 is a non-volatile memory that stores programs necessary for the CPU 151 to execute processing. Note that the program may be obtained via a recording medium 157 and an I / F unit 158 described below. The RAM 153 functions as a main memory and a work area of the CPU 151. That is, the CPU 151 loads a necessary program from the ROM 152 into the RAM 153 at the time of executing the processing, and realizes various controls and arithmetic processing by executing the program. In the case of the second embodiment, the CPU 151 executes display control on the image processing unit 105, the rewriting processing unit 114, the face detection unit 113, and the display unit 154 of the digital camera 100 in FIG. Each processing similar to the above is performed.

  The I / F unit 158 is an interface for connecting to an external device or the like by wireless or wired communication, or for connecting to an external memory or the like. An external device connected to the I / F unit 158 includes a digital camera, and a connection destination of the I / F unit 158 includes a network such as the Internet. Under the control of the CPU 151, the recording medium 157 records and reads information such as an image after rewriting processing, a virtual light source state image, and the like, such as a captured image acquired through the I / F unit 158 and additional information thereof.

  The display unit 154 includes a monitor such as a liquid crystal display (LCD). Under the control of the CPU 151, the display unit 154 displays an image after rewriting processing, a virtual light source state image, and the like, which are similar to those displayed on the display unit 109 of the digital camera 100 described above. The operation unit 156 includes, for example, a keyboard, a mouse, a touch panel, and the like, acquires an instruction from a user, and transmits the instruction to the CPU 151. The user can give an instruction to the virtual light source state image described in the first embodiment via the operation unit 156.

  The PC 150 according to the second embodiment generates a virtual light source state image in which parameters such as a position, an irradiation range, and an intensity of the virtual light source are schematized, similarly to the first embodiment described above. Is superimposed on the image of. In the second embodiment, as described above, the user can adjust the position, irradiation range, and intensity of the virtual light source based on the virtual light source state image. Also in the second embodiment, basic processing such as generation and superimposed display of a virtual light source state image, and adjustment of the position, irradiation range, and intensity of the virtual light source via the virtual light source state image are performed in accordance with the first processing described above. This is almost the same as the embodiment. In the following description, only points different from the first embodiment described above in the second embodiment will be described.

  FIG. 9 shows a flowchart of a process executed by the CPU 151 of the PC 150 according to the second embodiment. In the flowchart of FIG. 9, in the second embodiment, from the execution of the rewriting process on the captured image to the generation of a virtual light source state image indicating the positional relationship between the virtual light source and the subject, and the superimposed display on the image after the relighting process 3 shows the flow of processing.

In step S <b> 901, the CPU 151 reads out image data of a captured image that has been acquired from a digital camera or the like in advance and recorded on the recording medium 157.
In the next step S902, the CPU 151 determines whether or not an instruction to execute the rewriting process has been issued from the user via the operation unit 156. If the CPU 151 determines that the instruction to perform the rewriting process has not been issued by the user, the CPU 151 ends the process of the flowchart in FIG. 9, while determining that the instruction to perform the rewriting process has been issued. In this case, the process proceeds to S903.

  In step S903, the CPU 151 determines whether or not information related to face detection, that is, information on a face detection result, is included as additional information of the captured image acquired from the digital camera. If the CPU 151 determines that the additional information includes the face detection result, the process proceeds to S904. If the CPU 151 determines that the additional information does not include the face detection result, the process proceeds to S907.

If the processing has proceeded to S907, the CPU 151 executes face detection processing similar to the processing in the face detection unit 113 of the digital camera 100 of the first embodiment on the captured image.
Next, in step S908, the CPU 151 obtains luminance information in the face area included in the captured image by performing the same processing as that performed by the shadow obtaining unit 208 according to the first embodiment. Specifically, the CPU 151 acquires information such as an average luminance value in the face area and a luminance distribution such as a luminance histogram based on the face detection result.
Further, in step S909, the CPU 151 acquires luminance information outside the face area included in the captured image by performing the same processing as that of the above-described shadow acquisition unit 208. Specifically, the CPU 151 acquires information such as an average luminance value outside the face area and a luminance distribution such as a luminance histogram based on the face detection result. After S909, the CPU 151 proceeds to S910.

If the processing has proceeded to S904, the CPU 151 acquires information on the face detection result from the additional information of the captured image.
Next, in step S905, the CPU 151 acquires luminance information in the face area of the captured image based on the face detection result of the additional information.
Further, in step S906, the CPU 151 acquires luminance information outside the face area of the captured image based on the face detection result of the additional information. After this S906, the CPU 151 advances the processing to S910.

  In step S910, the CPU 151 determines whether the shooting target scene (scene) is a rewriting target scene that is effective when the rewriting process is performed in the same manner as described in step S606 of FIG. . If the CPU 151 determines that the scene is a rewriting target scene, the process proceeds to S911, and if it is determined that the scene is not a rewriting target scene, the process of the flowchart in FIG. 9 ends.

When the processing proceeds to S911, the CPU 151 performs the same processing as the distance calculation unit 303 described in the first embodiment based on the distance information acquired as the additional information of the captured image, and generates the weighted map image (mapK). Generate information.
Further, in S912, the CPU 151 performs a process similar to that of the normal line calculation unit 304 described above to generate a normal line map image (mapN).

Next, in step S <b> 913, the CPU 151 determines the reflection component calculation unit 305 described in the first embodiment based on the coordinate information of the face based on the face detection result and the luminance information inside and outside the face area described above. By the same processing, the position, irradiation range, and intensity of the virtual light source are determined.
Further, in S914, the CPU 151 generates a weighted map image (mapL) using the virtual light source by the same process as the reflection component calculation unit 305 of the first embodiment, and calculates the virtual light source reflection component in S915. .

Then, in the next S916, the CPU 151 generates a virtual light source state image similar to that described in the first embodiment.
Further, in S917, the CPU 151 executes the same relighting processing as described in the first embodiment, and in the next S918, in Step S918, displays the virtual light source state image superimposed on the image after the relighting processing. It is displayed on the unit 154.

  Thereafter, in S919, the CPU 151 determines whether or not to change the virtual light source state image by an operation from the user via the operation unit 156. If the CPU 151 determines that the change is to be made in response to an operation from the user, the process returns to step S914. If the CPU 151 determines that the change is not to be made, the process of the flowchart in FIG. 9 ends.

  As described above, the PC 150 according to the second embodiment acquires a captured image and additional information from a digital camera or the like, generates a virtual light source state image in which the parameters of the virtual light source are schematized based on the captured image and the additional information, and Alternatively, the image is superimposed on the image after the rewriting process. Thus, in the second embodiment, similarly to the first embodiment, the user can easily adjust each parameter such as the position, the irradiation range, and the intensity of the virtual light source based on the virtual light source state image superimposed and displayed. As a result, a desired relighting processing result can be obtained.

  In the above-described first and second embodiments, an example has been described in which the relighting process is performed using the weighted map image based on the distance added to the captured image and the weighted map image based on the normal, but the present invention is not limited to this. Absent. For example, according to the reliability of the distance information and the normal line information, the configuration may be such that the weighted map image based on the distance and the weighted map image based on the normal line are shaped and regenerated.

  In the above-described embodiment, an example has been described in which an image captured by a digital camera is subjected to relighting processing. However, for example, an image captured from an industrial camera, a vehicle-mounted camera, a medical camera, or the like, such as a smartphone or a tablet terminal, is subjected to relighting processing. The case is also applicable.

  A program for realizing one or more functions in signal processing according to the present invention can be supplied to a system or an apparatus via a network or a storage medium, and is read and executed by one or more processors of a computer of the system or the apparatus. It can be realized by doing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  Each of the above-described embodiments is merely an example of an embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

  100: digital camera, 50: system control unit, 105: image processing unit, 109: display unit, 113: face detection unit, 114: rewriting processing unit, 120: operation unit, 123: distance measurement sensor

Claims (15)

Acquiring means for acquiring a photographed image of a subject;
Setting means for setting a virtual light source;
Correction means for correcting the brightness of a specific subject in the captured image using the virtual light source,
Generating means for generating a virtual light source state image representing a positional relationship between the specific subject and the virtual light source,
Control means for superimposing and displaying the virtual light source state image on the captured image or the image after the brightness correction,
An image processing apparatus comprising:   The generating means may include a three-dimensional diagram or a plan view in which the positional relationship between the specific subject and the virtual light source is schematically illustrated as the virtual light source state image superimposed on the captured image or the image after the brightness correction. The image processing apparatus according to claim 1, wherein the image is generated. The setting means sets the position of the virtual light source, the irradiation range of the virtual light source, and the intensity of the virtual light source as parameters representing the virtual light source,
3. The virtual light source state image according to claim 1, wherein the generation unit generates the virtual light source state image representing a positional relationship between the specific subject and the virtual light source based on a parameter representing the virtual light source. Image processing device.   The setting unit includes a parameter indicating the virtual light source based on the position of the specific subject and luminance information in the specific subject area or luminance information in and outside the specific subject area. The image processing apparatus according to claim 3, wherein is set.   The image processing apparatus according to claim 3, wherein the generation unit sets a parameter representing the virtual light source based on information added to the captured image. The setting means, when an instruction to change the virtual light source state image from the user is input, based on the virtual light source state image changed by the instruction, reset the parameters,
The image processing apparatus according to claim 3, wherein the correction unit corrects brightness of the specific subject in the captured image based on the reset parameters. The setting means, when setting the parameters, calculates a range of parameters that can be set,
The image processing apparatus according to claim 3, wherein the control unit adds and displays the range of the parameter as a guideline to the virtual light source state image.   The method according to claim 3, wherein the setting unit sets one or a plurality of virtual light sources and, when the number of the virtual light sources is plural, sets the parameter for each virtual light source. 8. The image processing device according to any one of items 7 to 7. The setting means determines whether or not the specific subject is a target for performing brightness correction,
The image processing apparatus according to claim 1, wherein the correction unit corrects the brightness of the subject when it is determined that the subject is to perform the brightness correction. .   The setting means determines whether or not the specific subject is a target to be subjected to brightness correction based on luminance information within the specific subject area or luminance information within and outside the specific subject area. The image processing apparatus according to claim 9, wherein the determination is made.   The control unit, when it is determined that the specific subject is a target for which brightness correction is to be performed, superimposes the brightness correction recommendation mark on the photographed image or the image after the brightness correction. The image processing apparatus according to claim 9, wherein:   The image processing apparatus according to claim 1, further comprising a detection unit configured to detect a face area as the specific subject from the captured image. Photographing means for photographing a subject to generate a photographed image;
An image processing apparatus according to any one of claims 1 to 12,
An imaging device comprising: An image processing method executed by the image processing device,
An acquisition step of acquiring a photographed image of a subject,
A setting step of setting a virtual light source,
A correction step of correcting the brightness of a specific subject in the captured image using the virtual light source,
A generation step of generating a virtual light source state image representing a positional relationship between the specific subject and the virtual light source,
A control step of superimposing and displaying the virtual light source state image on the captured image or the image after the brightness correction,
An image processing method comprising:   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images