JP2018010497A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply adjust a position and light intensity of a virtual light source for relighting.SOLUTION: A camera signal processing unit 108 stores an image captured by an imaging sensor 109 in a DRAM 106. A subject detection unit 111 detects a subject to be subjected to relighting, from the captured image. A control microcomputer 101 displays the detected subject and an operation object to operate a position and light intensity of a virtual light source with respect to the subject, on an LCD of an LCD touch panel 116. A user operates the operation object by means of a touch panel of the LCD touch panel 116. The control microcomputer 101 controls a relighting processing unit 112 to execute relighting processing on the subject with the virtual light source, in response to an operation on the operation object, and displays a result of execution on the LCD.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that corrects the brightness of an input image.

  Conventionally, there has been known a relighting process for correcting the brightness of an image so that a subject in a captured image is irradiated with light from a virtual light source (virtual light source). By this processing, the brightness of the captured image can be partially corrected so that the shadow of the subject caused by the ambient light becomes more desirable.

  Patent Document 1 describes an image processing apparatus that determines the arrangement of a virtual light source with respect to ambient light according to the shadow state of a subject in a captured image and performs relighting processing.

JP 2006-072692 A

  In general, what kind of shading should be applied to the subject differs depending on the user. That is, when the arrangement of the virtual light source is automatically determined from the shaded state of the subject, a result that is desirable for many people may be different from an expectation for a certain user.

  An object of the present invention is to present an image processing apparatus that allows a user to easily adjust the shadow state of a subject.

  An image processing apparatus according to the present invention includes an input unit that inputs an image, a subject detection unit that detects a subject to be relighted from the image, a relighting processing unit that performs a relighting process on the subject using a virtual light source, A display means for displaying an operation object for operating the position and light intensity of the virtual light source with respect to the subject together with the subject, an operation means for operating the operation object, and a relighting processing means according to an operation of the operation object. Control means for executing the relighting process and updating the display of the display means.

  According to the present invention, the user can easily adjust the shadow state of the subject with a simple operation.

It is a schematic block diagram of one Example of this invention. It is an external appearance perspective view of a present Example. It is an example of an image of a relighting process. It is explanatory drawing of the positional relationship of the to-be-processed object and a virtual light source. It is an example of a screen which operates the position and light intensity of the virtual light source of a relighting process. It is explanatory drawing of the relationship between a horizontal slide bar and the horizontal position of a virtual light source. It is explanatory drawing of the template image used for the preview image generation of a relighting process. It is explanatory drawing of preview image generation. It is a flowchart of the relighting process of a present Example. It is an example of a screen which operates the position and light intensity of a virtual light source according to the inclination of the subject. It is an example of a screen for operating the position and light intensity of a virtual light source for a plurality of subjects. It is an example of a screen which operates the position and light intensity of a virtual light source which set a some virtual light source. It is a figure of another example of a screen which operates the position and light intensity of a virtual light source with a circular object, and description.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of a digital camera that employs an embodiment of an image processing apparatus according to the present invention.

  In the digital camera 100, reference numeral 101 denotes a microcomputer (hereinafter referred to as “control microcomputer”) that controls each block connected by the internal bus 102.

  Reference numeral 103 denotes a flash memory controller that controls reading / writing of data from / to the flash memory 104 based on a request from the control microcomputer 101. The control microcomputer 101 controls the digital camera 100 by reading and executing a program stored in the flash memory 104.

  Reference numeral 105 denotes a DRAM controller that controls reading / writing of data from / to the DRAM 106 based on a request from a predetermined block including the control microcomputer 101. A predetermined block including the control microcomputer 101 stores temporary data or image data during operation in the DRAM 106.

  An operation unit 107 includes a shutter button, a playback button, and the like provided on the exterior of the digital camera 100. The operation unit 107 transmits a user operation to the control microcomputer 101.

  Reference numeral 108 denotes a camera signal processing unit. The camera signal processing unit 108 processes an image signal captured by the image sensor 109 based on an instruction from the control microcomputer 101 and stores the obtained image data in the DRAM 106. The camera signal processing unit 108 and the imaging sensor 109 function as an image input unit that inputs image data to the image processing apparatus. The camera signal processing unit 108 also scales the image data to a size designated by the control microcomputer 101 and stores it in the DRAM 106 as recording image data. The camera signal processing unit 108 also scales the image data into display image data and stores it in the display frame memory of the DRAM 106. The camera signal processing unit 108 also synthesizes two pieces of image data at a ratio specified by the control microcomputer 101 and stores the obtained synthesized image data in the DRAM 106.

  Reference numeral 110 denotes a lens unit. The lens unit 110 includes a zoom lens, a focusing lens, and a diaphragm, and can adjust the lens position and the diaphragm based on instructions from the control microcomputer 101. Light from the subject received through the lens unit 110 enters the light receiving surface of the image sensor 109.

  Reference numeral 111 denotes a subject detection unit. The subject detection unit 111 detects human faces from the image data stored in the DRAM 106, detects their positions and sizes, and positions of organs such as eyes and mouth, and transmits them to the control microcomputer 101.

  Reference numeral 112 denotes a relighting processing unit. The relighting processing unit 112 sets a virtual light source for the image data stored in the DRAM 106 based on an instruction from the control microcomputer 101, and stores the image data obtained as a result of the relighting process for the specified subject in the DRAM 106.

  Reference numeral 113 denotes a memory card controller. The memory card controller 113 records the recording image data stored in the DRAM 106 on the memory card 150 connected to the digital camera 100.

  Reference numeral 114 denotes a graphic drawing unit. Based on an instruction from the control microcomputer 101, the graphic drawing unit 114 draws character information to be notified to the user and objects such as icons used for operation in the display frame memory of the DRAM 106.

  Reference numeral 115 denotes an LCD (Liquid Crystal Display) controller. The LCD controller 115 reads the data stored in the display frame memory of the DRAM 106 and supplies it to the LCD unit of the LCD touch panel 116 for display. 117 is a touch panel controller. The touch panel controller 117 detects a user's touch operation on the touch panel of the LCD touch panel 116 and transmits the touch position and the like to the control microcomputer 101.

  2A and 2B are external views of the digital camera 100. FIG. 2A is an external perspective view of the digital camera 100 viewed from the front side, and FIG. 2B is an external perspective view of the digital camera 100 viewed from the back side. Show. On the day of the week shown in FIG. 2A, the digital camera 100 includes a lens unit 110 on the front surface and a shutter button 201 constituting the operation unit 107 on the upper surface. As shown in FIG. 2B, an LCD touch panel 116 and a playback button 202 constituting the operation unit 107 are disposed on the back of the digital camera 100.

  The outline of the relighting process will be described with reference to FIGS. FIG. 3A shows an example of an image before the relighting process for the person 300, and FIG. 3B shows an example of an image after the process.

  In the pre-processing image 302 shown in FIG. 3A, the left part 304 of the face of the person 300 is brightened by the ambient light, and the right part 306 is relatively dark. Here, in order to simplify the figure, the light portion (left portion 304) and the dark portion (right portion 306) are shown separately, but in reality, the brightness changes continuously according to the surface shape of the face. To do.

  In the post-processing image 310 shown in FIG. 3B, a virtual light source is arranged on the opposite side of the ambient light with respect to the person 300. With the virtual light source, the right portion 306 becomes brighter than before processing, and the left portion 304 has a portion 312 that is slightly darker. Note that a portion that becomes darker than before processing can be formed by gain adjustment for preventing the bright portion from being overexposed by the addition of the brightness by the virtual light source. The portion 314 under the chin that is shaded from either the ambient light or the virtual light source is darkened.

  As a premise of the relighting process, it is necessary to determine the unevenness of the face. The control microcomputer 101 estimates the surface shape in the depth direction of the face based on the position and size of the face of the person 300 detected by the subject detection unit 111 and the positions of organs such as eyes and mouth. Alternatively, the distance in the depth direction of the portion to be processed can be detected or estimated by distance measuring means such as a phase difference detection method at the time of image capture, and the uneven shape can be determined using this distance data.

  For example, as shown in FIG. 4, the surface shape in the depth direction of the horizontal position indicated by rectangles 410 to 429 with respect to the face of the person 300 is estimated as a curve 420. The control microcomputer 101 obtains normal vectors 450 to 469 of the surface portions corresponding to the rectangles 410 to 429 in the curve 420 thus obtained. The control microcomputer 101 sets the center 470 of the face of the person 300 at a position where a predetermined distance is provided in the depth direction from the center surface of the curve 420. It is assumed that the virtual light source 471 is disposed at a sufficiently long distance from the center 470 in FIG. A vector 472 from the virtual light source 471 toward the center 470 indicates the optical axis of a virtual light beam emitted from the virtual light source 471. When the angle formed by the normal vectors 450 to 469 of the surface portions corresponding to the respective rectangles 410 to 429 and the vector 472 is within the range of 90 degrees to 270 degrees, the control microcomputer 101 corrects the rectangular area closer to 180 degrees brighter. To do. The control microcomputer 101 performs these processes on the entire subject to be relighted in units of pixels.

  FIG. 5 shows an example of a screen serving as a user interface for changing the position and intensity of the virtual light source. FIG. 5A shows an example of an image displayed on the LCD of the LCD touch panel 116 before the relighting process, and FIG. 5B shows an example of an image additionally displaying elements for operating the virtual light source.

  When the user presses the playback button 202, the control microcomputer 101 sets the digital camera 100 in the playback mode, and plays back and displays the image data stored in the memory card 150 on the LCD of the LCD touch panel 116. FIG. 5A shows an example image 500 reproduced and displayed on the LCD in this way. In the example image 500, a window 502 is in front of the right side of the person 300, and the ambient light from there illuminates the left side of the face of the person 300 brightly. The control microcomputer 101 displays a gear-like icon 504 on the upper left of the screen by the graphic drawing unit 114. When the user touches this icon 504, the control microcomputer 101 displays a predetermined menu, and the user can select a relighting process on the menu.

  When the user selects the relighting process on the menu, the control microcomputer 101 additionally displays a frame 506 indicating a face portion of a person as a processing target by subject detection as in an image example 505 illustrated in FIG. To do. The control microcomputer 101 additionally displays a horizontal slide bar 508 and a knob 510 for operating the position of the virtual light source in the horizontal direction, and a vertical slide bar 512 and a knob 514 for operating the output light intensity of the virtual light source in the vertical direction. The control microcomputer 101 further displays a cancel button 516 on the upper left of the screen and a determination button 518 on the upper right of the screen. The user can change the horizontal position of the virtual light source by sliding left and right while touching the knob 510 with respect to the touch panel of the LCD touch panel 116, and can increase the strength of the virtual light source by sliding up and down while touching the knob 514. Can be changed.

  The relationship between the position of the knob 510 on the horizontal slide bar 508 and the position of the virtual light source will be described with reference to FIG. FIG. 6A shows the movable range 601 of the virtual light source 471 with respect to the operation of the knob 510 of the slide bar 508. In response to the operation of the knob 510, the control microcomputer 101 moves the virtual light source 471 on the circumference of a certain distance R from the target center 470. However, the control microcomputer 101 always maintains the virtual light source 471 so as to face the center 470. As shown in FIG. 6B, each range obtained by equally dividing the entire length H of the horizontal slide bar 508 corresponds to each range obtained by equally dividing the movable range angle θ.

  A method of generating a preview image for relighting processing will be described with reference to FIGS. FIG. 7 shows an example of a preview image, and FIG. 8 shows a synthesis process of an image before processing and two temporary images.

  In the user interface of the relighting process, the control microcomputer 101 displays the result of changing the position and intensity of the virtual light source following the movement of the knob of the slide bar on the LCD of the LCD touch panel 116. That is, the control microcomputer 101 causes the relighting processing unit 112 to perform relighting processing of the position and light intensity of the virtual light source according to the operation of the slide bar knob. Then, the control microcomputer 101 updates the display on the LCD with the result of the relighting process or a preview image described below.

  The relighting process may take time with respect to the operation speed for the position of the knob. In the present embodiment, in order to cope with this, the control microcomputer 101 generates a preview image obtained by combining a plurality of previously created temporary images and displays it first.

  An image 701 shown in FIG. 7A shows an image before the relighting process is performed. P0 is one of the pixels constituting the image 701. An image 702 shown in FIG. 7B is a first temporary image obtained by performing the relighting process with the virtual light source set at the leftmost position 712 of the movable range 601. P1 is a pixel corresponding to the pixel P0 in the image 701 before processing. An image 703 shown in FIG. 7C is a second temporary image that has been relighted with the virtual light source set at the rightmost position 713 of the movable range 601. P2 is a pixel corresponding to the pixel P0 of the image 701 before processing.

FIG. 8 shows a composition relationship between the pre-processing image 701 and the two temporary images 702 and 703. The control microcomputer 101 first generates a temporary preview image 704 by combining the two temporary images 702 and 703 with the camera signal processing unit 108 or the relighting processing unit 112. Pt is a pixel of the temporary preview image 704 corresponding to the pixel P0 of the image 701 before processing. The pixel value pt of the pixel Pt is obtained by Expression 1 according to the position a of the knob 510 of the horizontal slide bar 508. That is,
pt = a * p1 + (1-a) * p2 (1)
Here, p1 and p2 are pixel values of the pixels P1 and P2, respectively. a changes in proportion to the position of the knob 4610, and becomes 0 when positioned at the left end of the horizontal slide bar 508, and becomes 1 when positioned at the right end.

Further, the control microcomputer 101 causes the camera signal processing unit 108 or the relighting processing unit 112 to combine the generated temporary preview image 704 and the unprocessed image 701 to generate a preview image 805. P in the preview image 805 is a pixel corresponding to the pixel P0 of the image 701 before processing. The pixel value p of the pixel P is obtained by Expression 2 according to the position b of the knob 514 of the vertical slide bar 512. That is,
p = b * pt + (1-b) * p0
= B * (a * p1 + (1-a) * p2) + (1-b) * p0 (2)
Here, b changes in proportion to the position of the knob 514, and becomes 0 when positioned at the upper end of the vertical slide bar 512, and becomes 1 when positioned at the lower end. By performing this synthesis process on all the pixels of the subject to be relighted, a preview image that follows the movement of the slide bar knob is generated.

  Temporary images are generated for a plurality of positions of the virtual light source, but temporary images are generated for two light intensities, a temporary preview image and a preview image are generated in the same procedure, and used for display during operation of the virtual light source. It may be.

  FIG. 9 shows a flowchart of the relighting process of this embodiment. The control microcomputer 101 implements the processing shown in FIG. 9 by reading the control program corresponding to FIG. 9 from the flash memory 104 and executing it. As described with reference to FIG. 4, when the user selects the relighting process from the menu display during the image reproduction, the control microcomputer 101 executes the process shown in FIG.

  In step S <b> 901, the control microcomputer 101 causes the subject detection unit 111 to detect the face of a person from the image being reproduced, and determines a main subject to be rewritten.

  In step S902, the control microcomputer 101 determines whether the main subject has been determined in step S901. If the main subject has been determined, the control microcomputer 101 proceeds to step S904, and if not, proceeds to step S903.

  In step S <b> 903, the control microcomputer 101 displays an interruption message on the LCD of the LCD touch panel 116. When the user touches the touch panel of the LCD touch panel 116, the control microcomputer 101 clears the display of the interruption message and ends the relighting process shown in FIG.

  In step S904, the control microcomputer 101 generates two temporary images as described with reference to FIGS. In step S <b> 905, the control microcomputer 101 causes the graphic drawing unit 114 to draw an operation object in the display frame memory of the DRAM 106. The drawing object includes operation objects such as a frame 506 surrounding the main subject, a horizontal slide bar 508 and a knob 510, a vertical slide bar 512 and a knob 514, and a determination button 518 and a cancel button 516.

  In step S906, the control microcomputer 101 starts generating a recording image. Specifically, the control microcomputer 101 starts a relighting process corresponding to the current position of the knob of the slide bar on the image before processing.

  In step S907, the control microcomputer 101 generates a preview image 705 by synthesizing the pre-processing image 701 and the two temporary images 702 and 703 as described with reference to FIGS. Display on LCD.

  In step S908, the control microcomputer 101 determines whether or not the determination button 518 has been touched. If the determination button 518 is touched, the control microcomputer 101 proceeds to step S909. If not, the control microcomputer 101 proceeds to step S912.

  In step S909, the control microcomputer 101 waits for the completion of the recording image generation started in step S906. If completed or already completed, the control microcomputer 101 proceeds to step S910. In step S <b> 910, the control microcomputer 101 records the generated recording image on the memory card 150. In step S911, the control microcomputer 101 performs an end process for deleting the generated temporary image and preview image, the drawn operation object, and the like to return to the state before the start of the relighting process, and ends the process shown in FIG.

  In step S912, the control microcomputer 101 determines whether or not the interruption button has been touched. If the interrupt button is touched, the control microcomputer 101 proceeds to step S913. If not, the control microcomputer 101 proceeds to step S914.

  In step S913, the control microcomputer 101 confirms whether or not the generation of the recording image is continuing. If it continues, the generation is interrupted, and the process proceeds to step S911. In step S911, the control microcomputer 101 executes an end process as described above, and ends the process shown in FIG.

  In step S914, the control microcomputer 101 determines whether or not the position of the slide bar knob has changed. If there is a change in the position of the knob, the control microcomputer 101 proceeds to step S915. If there is no change, the control microcomputer 101 proceeds to step S916.

  In step S915, as in step S913, the control microcomputer 101 confirms whether or not the generation of the recorded image is continuing. If the recording image is continued, the control microcomputer 101 interrupts the generation and returns to step S906.

  In step S916, the control microcomputer 101 confirms whether the generation of the recording image is completed. When the generation of the recording image is completed and can be used for displaying the preview image, the control microcomputer 101 proceeds to step S917. If the generation of the recording image has not been completed or if the recording image has already been used for display, the control microcomputer 101 returns to step S908.

  In step S917, the control microcomputer 101 replaces the image to be displayed on the LCD of the LCD touch panel 116 with a recording image from the preview image, and returns to step S908.

  As described above, according to the present embodiment, the user can adjust the shadow state of the subject with a simple operation.

  The main subject to be rewritten may be tilted. A second embodiment corresponding to such an inclination will be described.

  FIG. 10 shows a screen configuration example of the user interface according to the inclination of the main subject. When the user selects the relighting process from the menu, the control microcomputer 101 displays a screen as shown in FIG. 10 on the LCD of the LCD touch panel 116. The image to be processed here is different from the display example in FIG. 5B in that the face of the person 300 to be rewritten is tilted with respect to the LCD screen of the LCD touch panel 116.

  The control microcomputer 101 displays a frame 1006 covering the face portion on the main subject 300 in accordance with the inclination of the main subject 300. The control microcomputer 101 also displays the horizontal slide bar 1008 below the frame 1006 and the vertical slide bar 1012 on the right side in accordance with the inclination of the face portion. A knob 1010 for the horizontal slide bar 1008 and a knob 1014 for the vertical slide bar 1012 are movably disposed along the horizontal slide bar 1008 and the vertical slide bar 1012, respectively. The horizontal position of the virtual light source can be adjusted by the knob of the horizontal slide bar 1008, and the light intensity of the virtual light source can be adjusted by the knob 1014 of the vertical slide bar 1012, as in FIG. The movable range of the virtual light source is also tilted according to the tilt of the face portion.

  As described above, according to the second embodiment, the shadow state can be adjusted by irradiating the virtual light source from the left and right directions of the face according to the inclination of the face of the person in the image.

  A third embodiment will be described in which virtual light sources are set together or virtual light sources are set for each subject when there are a plurality of subjects to be relighted in one image.

  FIG. 11 shows an example of a screen displayed on the LCD of the LCD touch panel 116 in the third embodiment. In FIG. 11, in addition to the person 300, a person 1100 is included in one captured image. FIG. 11A shows a screen example of a batch application mode in which the relighting process using the virtual light source having the same setting is applied to both of the persons 300 and 1100, and FIG. 11B shows each of the persons 300 and 1100 individually. The example of a screen of individual application mode which sets a virtual light source is shown. For the person 1100, a frame 1106 surrounding the face portion is displayed in the same manner as the frame 506 for the person 300.

  In the collective application mode, the control microcomputer 101 displays an image as shown in FIG. 11A on the LCD of the LCD touch panel 116. That is, the control microcomputer 101 displays the frame 506, the horizontal slide bar 508 and the knob 510, and the vertical slide bar 512 and the knob 514 for one person 300 as in FIG. The user can adjust the position (angle) of the virtual light source with respect to the person 300 and the light intensity using the knobs 510 and 514. The control microcomputer 101 applies the same setting as the setting of the virtual light source for the person 300 to the other person 1100. Accordingly, the relighting process using the virtual light source having the same setting is applied to the two persons 300 and 1100. The functions of the cancel button 1116 and the enter button 1118 are the same as those of the cancel button 516 and the enter button 518, respectively.

  In the screen shown in FIG. 11A, a cross-shaped switching icon 1120 for instructing the control microcomputer 101 to switch to a mode in which virtual light sources are individually set is displayed at the upper left of the frame 1106 of another person 1100. . When the user taps the switching icon 1120, the control microcomputer 101 switches to the individual application mode of the screen example shown in FIG.

  In the individual mode, as shown in FIG. 11B, the control microcomputer 101 also displays a frame 1106 surrounding the face portion, a horizontal slide bar 1108 and a knob 1110, and a vertical slide bar 1112 and a knob 1114 on the person 1100. To do. The user can adjust the position (angle) and light intensity of the virtual light source with respect to the person 1100 separately from the virtual light source with respect to the person 300 using the knobs 1110 and 1114. The control microcomputer 101 draws a minus-shaped switching icon 1122 on the upper left of a frame 1106 that surrounds the face of the person 1100. When the user touches the icon 1122, the control microcomputer 101 switches to a screen in the batch application mode shown in FIG.

  In the third embodiment, it is possible to switch between applying a virtual light source setting to a plurality of human faces in an image at once or applying them individually with a simple operation.

  An embodiment 4 in which a plurality of virtual light sources can be set for one main subject will be described. FIG. 12 shows an example of a screen serving as the user interface. FIG. 12A shows an example of an initial screen when the relighting process is selected, and FIG. 12B shows an example of a screen that displays an operation element for operating the second virtual light source.

  In the initial screen shown in FIG. 12A, the control microcomputer 101 displays a frame 506, a horizontal slide bar 508 and a knob 510, and a vertical slide bar 512 and a knob 514 for the person 300 as in the example shown in FIG. To do. Displayed in the same manner as the cancel button 516 and the enter button 518. In this embodiment, the control microcomputer 101 displays an add icon 1220 for instructing addition of a virtual light source. When the user touches the add icon 1220, the control microcomputer 101 sets the horizontal slide bar 1238 and the knob 1240 and the vertical slide bar 1242 and the knob 1244 for the second virtual light source as shown in the screen example of FIG. indicate. Further, the control microcomputer 101 also displays a virtual light source deletion icon 1222. When the user touches the virtual light source deletion icon 1222, the control microcomputer 101 deletes the virtual light source corresponding to the knobs 510, 514, 1240, and 1244 selected at that time.

  In the fourth embodiment, a plurality of virtual light sources can be set to adjust the shadow state of the person's face.

  In each of the embodiments described above, the position and light intensity of the virtual light source are changed by the slide bar knob. However, the position or light intensity may be changed by an object that is an image of the virtual light source itself. FIG. 13A shows such a screen example 1301 of Example 5, and FIG. 13B shows a perspective view of the irradiation state of the virtual light source.

  The control microcomputer 101 displays an initial screen 1301 shown in FIG. 13A on the LCD of the LCD touch panel 116 when the relighting process is selected from the menu. Compared to the screen example shown in FIG. 5B, the frame surrounding the face of the person 300 and the slide bar are eliminated, and a circular object 1311 is added. The circular object 1311 indicates the position of the virtual light source at the center position, and indicates the light intensity at the radius. The user can change the position by touching and sliding the circular object 1311, and can change the radius (light intensity) by performing pinch-in or pinch-out in the vicinity of the circular object 1311. For example, the larger the radius of the circular object 1311, the stronger the output light intensity of the virtual light source. The color or radius and color of the circular object 1311 may be changed according to the output light intensity of the virtual light source.

  FIG. 13B shows the relationship between the display position of the circular object 1311 and the position of the virtual light source. Reference numeral 1320 denotes the center of the face of the person 300 on the display screen 1301. 1321 is ahead of the line segment Lo extended from the face center 1320 perpendicular to the display screen 1101 and indicates the position of the face center 1320 of the person 300 in the virtual depth direction. Reference numeral 1322 denotes a virtual light source (position). The virtual light source 1322 can be moved to an arbitrary position on a spherical surface having a radius R with the position 1321 as a center in consideration of the depth direction. Reference numeral 1323 denotes an intersection of a line segment La extending vertically from the virtual light source 1322 with respect to the display screen 1301 and the display screen 1301, and the circular object 1311 is drawn around the intersection 1323. The radius R is a fixed value, and the movable range of the circular object 1311 is limited to a range in which a spherical surface with the radius R centered on the position 1321 is projected on the display screen 1301. Of course, the radius R may be changed by the user's operation.

  According to the fifth embodiment, it is possible to adjust the shadow state of the person's face while easily grasping the position and strength of the virtual light source visually.

  Although the embodiment in which the position (or illumination angle) of the virtual light source can be adjusted in the horizontal plane has been described, it is obvious that the position in the vertical plane (elevation angle) can be changed by a similar interface.

  In each of the above embodiments, the face of a person is the target of the relighting process, but other types of subjects may be the target of the relighting process, or the entire image including the background may be the target of the relighting process. . The distance in the depth direction of the processing target portion is detected or estimated by distance measuring means such as a phase difference detection method at the time of image shooting, and the normal vector of the surface portion for each one or a predetermined number of pixel regions using this distance data Can be requested. By referring to this normal vector, brightness correction by a virtual light source can be executed as described above.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to the apparatus. At this time, the computer (or CPU or MPU) including the control unit of the supplied apparatus reads and executes the program code stored in the storage medium. The program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a magnetic disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, or a ROM can be used.

  In some cases, an OS (basic system or operating system) running on the apparatus performs part or all of the processing based on the above-described program code instructions, and the functions of the above-described embodiments are realized by the processing. included.

  Furthermore, the case where the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the apparatus or a function expansion unit connected to a computer, and the functions of the above-described embodiments are realized. It is. At this time, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

Claims (12)

An image input means for inputting an image;
Subject detection means for detecting a subject to be relighted from the image;
Relighting processing means for performing relighting processing on the subject by a virtual light source;
Display means for displaying, together with the subject, an operation object for manipulating the position and light intensity of the virtual light source with respect to the subject;
An operation means for operating the operation object;
An image processing apparatus comprising: a control unit that causes the relighting processing unit to execute the relighting processing and update the display of the display unit in response to an operation of the operation object.   The image processing apparatus according to claim 1, wherein the operation object includes a horizontal slide bar that slides in a horizontal direction and a vertical slide bar that slides in a vertical direction.   The image processing apparatus according to claim 2, wherein a position of the virtual light source in the horizontal plane with respect to the subject can be operated by the horizontal slide bar.   The image processing apparatus according to claim 2, wherein the light intensity of the virtual light source with respect to the subject can be operated by the horizontal slide bar.   The operation object is formed of a circular object, the center of the circular object indicates the position of the virtual light source with respect to the subject, and the radius of the circular indicates the light intensity of the virtual light source. The image processing apparatus described.   6. The image processing apparatus according to claim 1, further comprising a recording unit that records the image including the subject processed by the relighting processing unit. The subject detection means detects the inclination of the subject in the image;
The image processing apparatus according to claim 1, wherein the display unit displays the operation object at a position and a direction according to an inclination of the subject.   8. The control unit according to claim 1, wherein the control unit controls the relighting processing unit to apply a virtual light source for the main subject to another subject when there are a plurality of the subjects. 9. The image processing apparatus described.   The said display means displays the operation object which operates the position and light intensity of the said virtual light source with respect to each to-be-photographed object, when there are two or more said to-be-photographed objects. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the display unit can display an operation object of a plurality of virtual light sources on a subject. Furthermore,
Means for generating two temporary images resulting from a relighting process for two different positions of the virtual light source;
11. The image processing apparatus according to claim 1, further comprising: a unit that generates a preview image by combining the two temporary images in accordance with an operation of the virtual light source.   The image processing apparatus according to claim 1, wherein the image input unit is an imaging unit.

Images