JP6576083B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to a technique for adding a shadow to image data.

  When a subject is photographed using an imaging device, the image obtained varies greatly depending on how the light strikes the subject. For example, when photographing in a backlight condition, the whole or part of the subject becomes shadowed and appears dark. In addition, when shooting is performed by irradiating a subject with light using a strobe, shadows of the subject may be skipped due to the influence of light, and the subject may appear flat. As a method for correcting these images, there is known a method for correcting a dark part of a subject so that light is artificially applied from a specific direction (Patent Document 1). In the technique described in Patent Document 1, the influence of a virtual light source is expressed by a Gaussian distribution. At this time, by biasing the Gaussian distribution for luminance correction according to a separately set lighting direction, it is possible to obtain an image that is virtually illuminated from a desired direction.

  In addition, CG rendering processing is performed under a predetermined virtual illumination condition using a 3D model corresponding to the subject, and the lighting of the subject is changed by replacing the rendered CG image with the subject image in the image obtained by shooting. The method of doing is also known (patent document 2). In the technique described in Patent Document 2, a 3D model to be replaced with a subject is determined from 3D models prepared in advance.

Japanese Patent No. 5281878 Japanese Patent No. 5088220

  When another image as if it was taken under the lighting conditions desired by the photographer is generated from the taken image, it is preferable to obtain an image with a shadow that matches the posture and expression of the subject. However, the techniques of Patent Documents 1 and 2 have the following problems.

  When the influence of a virtual light source is expressed by a Gaussian distribution as in the technique described in Patent Document 1, it is difficult to give a natural shadow to a subject having a complicated shape such as a human face. In addition, when the subject image in the image is replaced with the result of rendering using the 3D model of the subject as in the technique described in Patent Document 2, the posture and expression of the subject after replacement are changed to the posture and expression of the 3D model. It will be replaced with an unnatural result. Therefore, an object of the present invention is to obtain an image to which a shadow according to the posture and expression of a subject is given.

In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus that generates a corrected image in which lighting for a subject is changed in a captured image obtained by capturing the subject, an image obtaining unit for obtaining a captured image, the distance from the imaging device imaging the subject to the subject, a distance obtaining means for obtaining distance information indicating for each pixel of the captured image, in order to generate the corrected image Setting means for setting illumination parameters related to the virtual light source, holding means for holding face model data (for example, normal image data described later) representing a three-dimensional shape of a predetermined face, and the face of the subject in the captured image Detecting means as a face area, and transforming the face model data based on the face area detected by the detecting means Thus, a lighting unit that corrects the pixel value of each pixel in the captured image using a deformation unit that generates corrected face data and a correction amount according to the corrected face data, the distance information, and the illumination parameter. And processing means for performing the processing.

  According to the present invention, it is possible to obtain an image to which a shadow according to the posture and expression of the subject is given.

1 is a diagram illustrating an external appearance of an imaging apparatus according to Embodiment 1 of the present invention. 1 is a diagram showing an internal configuration of an imaging apparatus according to Embodiment 1 of the present invention. 1 is a block diagram illustrating a configuration of an image processing unit according to a first embodiment of the present invention. The flowchart which shows the flow of the process which concerns on Embodiment 1 of this invention. FIG. 3 is a diagram illustrating an example of image data according to the first embodiment of the present invention. The figure which shows the face information which concerns on Embodiment 1 of this invention. The flowchart which shows the flow of the distance correction process which concerns on Embodiment 1 of this invention. The figure showing the outline | summary of the distance correction process which concerns on Embodiment 1 of this invention. 3 is a flowchart showing a flow of normal correction processing according to the first embodiment of the present invention. The figure showing the outline | summary of the normal correction process which concerns on Embodiment 1 of this invention. The figure showing the outline | summary of the normal-line smoothing process which concerns on Embodiment 1 of this invention. 5 is a flowchart showing a flow of lighting processing according to the first embodiment of the present invention. The figure which shows the outline | summary of the lighting process which concerns on Embodiment 1 of this invention. The figure showing the effect of the lighting process which concerns on Embodiment 1 of this invention. The figure which shows the example of the correction coefficient which concerns on Embodiment 2 of this invention. The figure showing the effect of the lighting process which concerns on Embodiment 3 of this invention. The flowchart which shows the flow of the process which concerns on Embodiment 4 of this invention.

[Embodiment 1]
<Appearance of imaging device>
FIG. 1 is a diagram illustrating an appearance of an imaging apparatus according to the present embodiment. FIG. 1A illustrates an appearance of the front surface of the imaging apparatus, and FIG. The imaging apparatus 101 includes an optical unit 102, an imaging button 103, a strobe 104, a distance acquisition unit 105, a display unit 106, and an operation button 107.

  The optical unit 102 is a lens barrel including a zoom lens, a focus lens, a shake correction lens, a diaphragm, and a shutter, and collects light information of a subject. The imaging button 103 is a button for the user to instruct the imaging apparatus 101 to start imaging. The strobe 104 is illumination that can emit light in accordance with the start of imaging in accordance with a user instruction. The distance acquisition unit 105 is a distance acquisition module that acquires distance image data of a subject in accordance with an imaging instruction. The distance image data means image data in which the subject distance corresponding to the pixel is stored as the pixel value of each pixel of the image. The distance acquisition unit 105 includes an infrared light emitting unit that emits infrared light and a light receiving unit that receives infrared light reflected by the subject, and determines the time until the emitted infrared light is reflected by the subject and received. Based on this, the distance value from the imaging device to the subject is calculated. Then, based on the calculated distance value and distance imaging information including the number of sensor pixels and the angle of view of the light receiving unit, the position information of the subject is calculated to generate distance image data. Note that the distance image data acquisition method is not limited to this. For example, instead of the distance acquisition unit 105, an optical system similar to the optical unit 102 is provided, and distance image data is acquired by performing triangulation based on parallax between image data captured from two different viewpoints. It may be.

  The display unit 106 is a display such as a liquid crystal display that displays image data processed by the imaging apparatus 101 and other various data. In the present embodiment, since no optical finder is provided in the imaging apparatus 101, a framing operation (confirmation of focus and composition) is performed using the display unit 106. That is, since imaging is performed while confirming the live view image on the display unit 106, it can be said that the display unit 106 functions as an electronic viewfinder while performing framing and focusing operations. On the display unit 106, live view display for displaying the imaging range in real time is performed, and a camera setting menu is displayed.

  The operation button 107 is a button for the user to instruct the imaging apparatus 101 to perform an operation mode switching operation of the imaging apparatus 101 and various parameters at the time of imaging. In the present embodiment, as one of operation modes, a lighting correction processing mode for correcting the degree of illumination in a captured image after imaging is included. The user can use the operation button 107 or the imaging button 103 to switch to the lighting correction processing mode, set lighting parameters for virtual lighting used for lighting correction, or select a subject for adjusting the lighting condition. it can. In addition, when outputting the corrected image data, the user can instruct whether or not to output distance image data. Note that the display unit 106 may have a touch screen function. In that case, a user instruction using the touch screen can be handled as an input of the operation button 107.

<Internal configuration of imaging device>
FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 101 according to the present embodiment.

  The CPU 202 is involved in all the processes of each component, and sequentially reads and interprets instructions stored in a ROM (Read Only Memory) 203 and a RAM (Rondom Access Memory) 204, and executes processes according to the results. The system bus 212 is a bus for transmitting and receiving data. In the present embodiment, it is assumed that the ROM 203 stores a face normal model corresponding to a human face. The face normal model indicates normal image data corresponding to a face having a predetermined shape, in which normal vectors of the face surface are stored as pixel values, and positions of human organs such as eyes and mouth in the normal image data. Organ location information.

  The control unit 206 is a control circuit that receives user instructions from the imaging button 103 and the operation button 107 and controls imaging, switching of a lighting correction processing mode, selection of a subject area, setting of illumination parameters, and the like. The optical system control unit 205 is a control circuit that performs control instructed by the CPU 202 such as focusing, opening a shutter, and adjusting an aperture to the optical unit 102.

  The color image sensor unit 201 is an image sensor that converts light information collected by the optical unit 102 into a current value. The color image sensor unit 201 includes a color filter having a predetermined arrangement such as a Bayer arrangement, and the color information of the subject is acquired from the light collected by the optical unit 102.

  The A / D conversion unit 208 is a processing circuit that converts the color information of the subject detected by the color image sensor unit 201 into a digital signal value and generates RAW image data. In this embodiment, it is assumed that distance image data and RAW image data captured at the same time can be acquired.

  An image processing unit 209 performs development processing on the RAW image data acquired by the A / D conversion unit 208 to generate color image data. The image processing unit 209 performs various image processing such as generating corrected image data obtained by performing lighting correction on the color image data using the color image data and the distance image data. The internal structure of the image processing unit 209 will be described in detail later.

  The character generation unit 207 is a processing circuit that generates characters, graphics, and the like. The characters and graphics generated by the character generation unit 207 are displayed on the display unit 106 while being superimposed on image data, corrected image data, and the like.

  The encoder unit 210 performs processing for converting various image data including color image data processed by the image processing unit 209 and corrected image data generated by lighting correction processing into a file format such as Jpeg.

  The media I / F 211 is an interface for transmitting / receiving image data to / from a PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.). As the media I / F 211, for example, a USB (Universal Serial Bus) is used.

<Internal configuration of image processing unit>
FIG. 3 is a block diagram showing a functional configuration of the image processing unit 209 in the present embodiment. The development processing unit 301 performs white balance processing, demosaic processing, noise reduction processing, color conversion processing, edge enhancement processing, gamma processing, and the like on the RAW image data acquired from the A / D conversion unit 208, and converts the color image data. Generate. The generated color image data can be output to the display unit 106 for display or stored in a storage device such as the RAM 204 or the PC / media 213. In the present embodiment, the development processing unit 301 generates color image data without performing gamma processing, and outputs the color image data to the lighting unit 305.

  The distance correction unit 302 generates correction distance data corresponding to the subject selected from the distance image data based on the color image data, the face information, and the subject position selected by the user. In the present embodiment, it is assumed that the correction distance data mainly stores a person corresponding to the selected subject position and other distance values corresponding to the background.

  The face detection unit 303 acquires the face information of the subject from the color image data acquired from the development processing unit 301. The face information of the subject includes at least information related to the face region indicating the region occupied by the face of the subject in the color image data and the organ position indicating the position in the color image data such as the eyes and mouth included in the face.

  The normal correction unit 304 corrects the face normal model stored in the ROM 203 based on the face information acquired from the face detection unit 303 and the color image data acquired from the development unit 301.

  The lighting processing unit 305 performs processing on the color image data based on the correction distance data acquired from the distance correction unit 302, the correction normal data acquired from the normal correction unit 304, and the illumination parameter acquired from the control unit 206. To write. The corrected image data generated by the writing process can be output to a storage device such as the RAM 204 or the PC / media 213 and stored, or output to the display unit 106 and displayed.

<Processing flow of image processing unit>
FIG. 4 is a flowchart showing an operation procedure of the image processing unit 209 in the imaging apparatus of the present embodiment. In the present embodiment, the image processing unit 209 uses the face information acquired from the color image data and the subject position P0 acquired based on the user instruction to calculate correction distance data corresponding to the selected subject from the distance image data. Generate. Then, based on the face information of the subject and the face normal model stored in advance, normal image data matching the face of the subject is generated. Thereafter, based on the illumination parameter set by the user operation, the correction distance data, and the generated normal image data, a lighting process for adding a virtual light source to the color image data is performed to generate corrected image data. Details of the operation procedure of the image processing unit 209 will be described below.

  In step S <b> 401, the development processing unit 301 performs development processing such as demosaic processing on the RAW image data acquired from the A / D conversion unit 208 to generate color image data. Color image data in the present embodiment will be described with reference to FIG. It is assumed that RGB values are stored as pixel values in the pixel (i, j) of the color image data I501, and are represented as Ir (i, j), Ig (i, j), and Ib (i, j), respectively. And Note that the color image data acquisition method is not limited to this. For example, RAW image data stored in the RAM 204 or the PC / media 213 may be acquired, and the development processing unit 301 may generate color image data. Alternatively, color image data that has already undergone development processing may be acquired from the RAM 204 or the PC / media 213. In step S <b> 402, the developing unit 301 outputs the color image data acquired in step S <b> 401 to the display unit 106. Based on the display on the display unit 106, the user determines whether to perform the lighting correction process.

  In step S <b> 403, the control unit 206 determines whether or not an instruction to perform lighting correction processing is input in accordance with an input from the operation unit 107. If an instruction to perform the lighting correction process has not been input, the process proceeds to step S404. If an instruction to perform lighting correction processing has been input, the control unit 206 outputs a signal indicating that lighting correction is to be performed to the developing unit 301 and the distance correction unit 302, and the process proceeds to step S406.

  In step S404, the control unit 206 determines whether an image output instruction is input by the user. If it is determined that the user has input an image output instruction, the process advances to step S405. If it is not determined that the user has input an image output instruction, the process returns to step S403.

  In step S405, the control unit 206 outputs an image output instruction to the developing unit 301, and the developing unit 301 outputs the color image data to the PC / media 213 and ends the processing.

  In step S <b> 406, the distance correction unit 302 acquires distance image data from the distance acquisition unit 105. The distance image data in this embodiment will be described with reference to FIG. It is assumed that the distance value D (i, j) from the imaging device to the subject is stored as a pixel value in the pixel (i, j) of the distance image data D502. Note that the distance image data acquisition method is not limited to this. For example, distance image data stored in the RAM 204 or the PC / media 213 may be acquired.

  In step S407, the developing unit 301 outputs color image data to the face detection unit 303, and the face detection unit 303 acquires face information of the subject from the input color image data. The face information in this embodiment will be described with reference to FIG. The face information in the present embodiment includes information indicating the face area 601 and the organ position 602. The face area represents a set of pixels in the area including the face in the color image data 501. The organ position 602 represents coordinates corresponding to the eyes and mouth in the face area. An existing algorithm can be applied to the detection method of the face area and the organ position. Examples include an algorithm using template matching and an algorithm using Haar-Like feature. In this embodiment, the face area / organ position is detected by template matching. First, a skin color region is extracted as a face candidate region by performing threshold processing on color image data. That is, a pixel whose pixel value falls within a range of pixel values determined based on various skin colors is extracted as a face candidate region. Then, matching processing is performed on the face candidate area using face image templates of various sizes, and the likelihood as the face area is calculated. Finally, an area where the calculated likelihood is equal to or greater than a predetermined threshold is extracted as a face area. In addition, the face detection unit 303 performs similar template matching on the extracted face area using the eye and mouth image templates, and extracts coordinates corresponding to the eyes and mouth. The face region 601 and the organ position 602 are acquired by the above processing. The face detection unit 303 outputs the acquired face information to the normal line correction unit 304. As an organ to be detected, other organs such as a nose and ears may be extracted in addition to the eyes and mouth.

  In step S408, the distance correction unit 302 determines the position of the subject designated by the user. In the present embodiment, the user uses the touch panel provided on the display unit 106 or the operation button 107 to specify the position of the subject on which the lighting correction process is to be performed. The distance correction unit 302 acquires the subject selection position P <b> 0 ′ input by the user operation from the control unit 206. Based on the acquired subject selection position P0 ', the designated subject position P0 in the color image data is calculated. In the present embodiment, color image data is displayed on the display unit 106 having a touch screen function, and an operation in which the user touches the subject on the display screen is accepted. The distance correction unit 302 determines the position touched by the user as the subject selection position. Obtained from the control unit 206 as P0 ′. At this time, the subject selection position P 0 ′ corresponds to the pixel position of the display unit 106. The distance correction unit 302 calculates the subject position P0 by converting the pixel position on the display unit 106 into the pixel position of the color image data.

  In step S409, the distance correction unit 302 generates correction distance data from the distance image data acquired in step S406 using the subject position P0 acquired in step S408 and the color image data acquired from the development unit 301. Details of the correction distance data generation processing will be described later. The distance correction unit 302 outputs the generated correction distance data to the writing unit 305.

  In step S410, based on the face information acquired by the normal correction unit 304 from the face detection unit 303 and the color image data input from the development unit 301, a correction method that is normal image data matched to the face of the subject. Generate line data. Details of the correction normal data generation processing will be described later. The normal correction unit 304 outputs the generated correction normal data to the lighting unit 305.

  In step S411, the lighting unit 305 performs a lighting process such as adding a virtual light source to the color image data based on the input correction distance data and correction normal data, and generates corrected image data. Details of the lighting process will be described later.

  In step S <b> 412, the lighting unit 305 determines whether a change in setting of illumination parameters used for the lighting process is input from the control unit 206. If it is determined that the setting of the illumination parameter has been changed, the process returns to step S411 and the lighting process is performed again. If it is determined that the setting of the illumination parameter has not been changed, the process proceeds to step S413.

  In step S 413, the writing unit 305 determines whether an image output instruction is input from the control unit 206. If it is determined that an image output instruction has been input, the process proceeds to step S414. If it is determined that an image output instruction has not been input, the process returns to step S412. In step S414, the lighting processing unit 305 outputs the generated corrected image data to the PC / media 213 and ends the process. The above is the flow of processing performed by the image processing unit 209 of the present embodiment. According to the above processing, the lighting process can be performed using the face normal model deformed according to the subject, so that an image with a natural shadow according to the posture and expression of the subject can be obtained. Details of processing performed by each component of the image processing unit 209 will be described below.

<Correction distance data generation processing>
Here, the correction distance data generation processing performed by the distance correction unit 302 in step S409 will be described with reference to the flowchart shown in FIG. In step S701, the distance correction unit 302 extracts a subject candidate area based on the face information, the subject position P0, and the distance image data. The process of this step will be described with reference to FIGS. First, the distance correction unit 302 selects the face area 601 closest to the subject position P0 from the face areas indicated by the face information. Then, the distance value of each pixel in the selected face area is acquired from the distance image data, and the average value thereof is calculated as the distance value of the face area. After that, the distance correction unit 302 generates a binary image 801 divided into pixels in which the difference in distance value from the distance value of the face area is equal to or less than a predetermined threshold and other pixels. In other words, the processing performed here is processing for discriminating a subject whose distance from the selected subject is included in a predetermined range and other subjects. Here, in the binary image 801, a pixel whose distance value difference from the face area distance value is equal to or smaller than a threshold value is set as a subject candidate area 802. The determination of the subject candidate area performed here is not limited to the above-described method, and an area where the difference in distance value from the selected subject position is within a predetermined threshold may be determined as the subject candidate area.

  In step S <b> 702, the distance correction unit 302 performs a small component removal process or a hole filling process on the binary image 801 to remove a small connected component included in the subject candidate area or perform a shaping process to fill a hole. As the small component removal process / hole filling process, a method using a morphological operation or a method using a labeling process can be applied. Here, a method using a morphological operation is used. The distance correction unit 302 performs an opening process on the subject candidate area included in the binary image 801 as the small component removal process. Then, as a subsequent hole filling process, a closing process is performed on the subject candidate area. FIG. 8C shows an example of a binary image 803 obtained by this step.

In step S703, the distance correction unit 302 performs a smoothing process on the binary image 803 subjected to the shaping process in step S702, and generates multi-value correction distance data 804 (FIG. 8D). For example, by performing a smoothing process for pixel values of pixels included in the subject candidate region 802 of the binary image 803 255, for other images to zero the pixel values of the pixel, 8 bits per pixel Correction distance data 804 having distance information is generated. At this time, the larger the pixel value, the smaller the distance to the subject.

  As the smoothing process, a Gaussian filter, a joint bilateral filter that performs smoothing while referring to pixel values of color image data, or the like can be applied. In the present embodiment, a joint bilateral filter represented by the following formula (1) is used.

  s is a pixel to be processed, Ω is a neighboring region of s, p is a pixel included in Ω, I is image data to be smoothed, R is reference image data, f is a weight based on the distance between p and s, g represents a weight based on the pixel value. f is set so that the weight decreases as the distance between s and p increases. g is set so that the weight decreases as the difference between the pixel values of the pixel p and the pixel s in the reference image increases. In Expression (1), Y represents the luminance difference between the pixel values of the pixel p and the pixel s. In step S703, smoothing processing is performed using correction distance data 803 as I and color image data as R. By using a joint bilateral filter with respect to the binary image 803 while referring to the color image data, it is possible to perform a smoothing process using only pixels having similar pixel values in the color image data. As a result, the contour of the subject area 802 can be smoothed while matching the contour of the subject in the color image data. Note that the smoothing method is not limited to this. For example, as a setting method of f, equal weights may be given in the neighborhood region. Further, as a setting method of g, a weight may be used based on a color difference instead of a luminance value. Alternatively, if the pixel value is within a certain value, the weight may be constant.

  Also, by acquiring the correction distance data 804 as a multi-valued image, it is possible to reduce the sense of incongruity of the subject outline during the lighting process performed in step S409. Through the above processing, the distance correction unit 302 can acquire correction distance data 804 that is mainly divided into the foreground subject and the other background, and corresponding distance values are stored. Note that the filter processing performed here does not have to be a joint bilateral filter, and any filter processing based on pixel values of color image data may be used.

<Normal image data generation processing>
Here, the correction normal data generation processing performed by the normal generation unit 304 in step S410 will be described. The correction normal data generation processing in the present embodiment is processing for correcting the face normal model stored in the ROM 203 or the PC / media 213 based on the color image data. Details of the correction normal data generation processing will be described below with reference to the flowchart shown in FIG.

  In step S901, a deformation parameter for deforming the face normal model according to the color image data is calculated. An example of face normal information of the present embodiment is shown in FIG. The face normal model includes face normal image data 1001 and organ position information 1002 corresponding thereto. The face normal image data 1001 stores face normal vectors (Nx (i, j), Ny (i, j), Nz (i, j)) as pixel values in the pixel N (i, j). Image data. Nx (i, j), Ny (i, j), and Nz (i, j) are the x-axis, y, which are three coordinate axes orthogonal to each other of the normal vectors stored in the pixel (i, j). It is a component in the axial and z-axis directions. Also, all normal vectors included in the face normal image data 1001 are unit vectors. For the pixels corresponding to the face area, vectors in the direction perpendicular to the face surface are stored as normal vectors, and for the pixels corresponding to the areas other than the face, vectors in the direction opposite to the optical axis of the imaging device are normal vectors. Is stored as In this embodiment, it is assumed that the z-axis is opposite to the optical axis of the imaging apparatus, and (0, 0, 1) is stored as a normal vector in pixels corresponding to regions other than the face. The organ position information 1002 indicates the coordinate values of the right eye, left eye, and mouth in the face normal image data 1001.

In this step, the normal line correction unit 304 calculates color image data 501 and face normal image data 1001 from the organ position 1002 corresponding to the face normal model and the organ position 602 included in the face information of the color image data. Associate the right eye, left eye, and mouth coordinates. Then, the normal correction unit 304 calculates a deformation parameter for matching the organ position 1002 of the face normal image data 1001 with the organ position 602. As the deformation parameter, an affine transformation coefficient to be used for affine transformation is calculated. As a method for calculating the affine transformation coefficient, a least square method or the like can be used. That is, an affine transformation coefficient that minimizes the sum of squares of errors from the organ position 602 when the organ position 1002 is affine transformed is determined as the transformation parameter here. In the present embodiment, the face normal image data 1001 has components in the x-axis, y-axis, and z-axis directions of normal vectors as pixel values. For example, each channel of 3-channel 8-bit color image data has these components. May be assigned. For example, since the components in the direction of each axis of the normal vector take values from −1.0 to 1.0, the value between these values is assigned from 0 to 255, so that the normal vector information is converted into 3-channel 8-bit color image data. Can be held.

  In step S902, the normal correction unit 304 converts the face normal image data 1001 using the affine transformation coefficient calculated in step S901, and generates normal image data 1003. As a result, normal image data 1003 is generated by fitting the face normal image data 1001 to the face area included in the color image data 501. The normal line image data 1003 includes normal vectors (N′x (i, j), N′y (i, j), N′z (i, j)) as pixel values for the pixel N ′ (i, j). ) Is stored. The normal vector of the normal image data 1003 is the normal vector (Nx, Ny, N) stored in each pixel of the face normal image 1001 for the region corresponding to the face normal image 1001 (FIG. 10B). Nz). For a region that does not correspond to the face normal image 1001, a normal vector (0, 0, 1) in the direction opposite to the optical axis of the imaging device is stored. By this step, the face area in the face normal image 1001 can be substantially matched with the face area in the color image data. However, positions other than the organ, such as the contour of the face, may not be accurately matched, and this is corrected in the subsequent steps.

  In step S903, the normal correction unit 304 divides the normal image data 1003 into components in the x-axis, y-axis, and z-axis directions, and generates x-axis component normal data 1101, y-axis component normal data 1102, and z-axis. The component normal data 1103 is decomposed into three image data (FIG. 11A). Thereby, the same smoothing process as that of the binary image 803 can be applied. In the present embodiment, the normal line correction unit 304 applies a joint bilateral filter as in step S703.

  In step S904, the normal correction unit 304 performs a smoothing process on the x-axis component normal data 1101 to generate smoothed x-axis component normal data 1104. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed x-axis component normal line data 1104 obtained by this processing stores a smoothed x-axis component value N ″ x for each pixel.

  In step S <b> 905, the normal correction unit 304 performs a smoothing process on the y-axis component normal data 1102 to generate smoothed y-axis component normal data 1105. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed y-axis component normal line data 1105 obtained by this processing stores the y-axis component value N ″ y smoothed in each pixel.

In step S <b> 906, the normal correction unit 304 performs a smoothing process on the z-axis component normal data 1103 to generate smoothed z-axis component normal data 1106. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed z-axis component normal line data 1106 obtained by this processing stores a smoothed z-axis component value N ″ z for each pixel.
Through steps S904 to S906, the contour of the face in the normal image data 1003 can be matched with the contour of the subject in the color image data.

  In step S907, the normal correction unit 304 integrates the smoothed x-axis component normal data 1104, the smoothed y-axis component normal data 1105, and the smoothed z-axis component normal data 1106 to obtain smoothed normal image data 1107. Is generated (FIG. 11B). The smoothed normal image data 1107 is an image in which normal vectors (N ″ x (i, j), N ″ y (i, j), N ″ z (i, j))) are stored in the pixel (i, j). It is data.

  In step S908, the normal vector stored in each pixel of the smoothed normal image data 1107 is normalized so as to become a unit vector. In steps S904 to S906, since the smoothing process is performed for each axis component, the size of the normal vector stored for each pixel differs. In order to correct this, in this step, normalization is performed so that the size of the normal vector becomes 1 as shown in Expression (2).

  Thus, a normal vector (N ′ ″ x (i, j), N ′ ″ y (i, j), N ′ ″ z (i, j)) of size 1 is applied to the pixel (i, j). ) Is stored.

  As described above, the normal correction unit 304 acquires correction normal data. According to the above processing, the face normal model can be corrected in accordance with the face of the subject, so that a natural shadow can be given to the face of the subject in the lighting processing. Further, by performing the smoothing process independently for each coordinate axis component as described above, it is possible to prevent the normal direction from being greatly changed by the smoothing process.

<Lighting processing>
Here, the lighting process performed in step S411 will be described. The lighting process according to the present embodiment is a process of generating a corrected image by performing a process of adding a virtual light source to color image data according to the illumination parameter set by a user operation based on the correction distance data and the correction normal data. It is. Details of the lighting process will be described below with reference to the flowchart shown in FIG.

  In step S <b> 1201, the lighting unit 305 acquires illumination parameters used for the lighting process set by the user from the control unit 206. In the present embodiment, the user sets the virtual illumination position Q, posture U, intensity α, and light source color L as illumination parameters by operating the operation unit 107.

  In step S1202, the lighting unit 305 corrects the pixel value of the color image data 501 based on the correction distance data 804, the normal image data 1003, and the illumination parameter acquired in step S1101. In the present embodiment, it is assumed that the pixel value of the color image data is corrected according to Expression (3), and I ′ is generated from the corrected image data.

Here, I′r, I′g, and I′b are the pixel values of the corrected image data I ′, Lrm, Lgm, and Lbm are the colors of the mth illumination, and km is the degree of correction of the pixel values for the mth illumination. To express. km is determined based on the brightness value α, the position Q, the posture U, the distance value corresponding to the pixel (x, y), and the normal vector N. For example, it can obtain | require like Formula (4).

  Equation (4) will be described with reference to FIG. t is a correction coefficient for adjusting the degree of correction by the virtual light source. In this embodiment, t = 1. α is a variable representing the brightness of illumination. Q is a vector representing the position of the light source. P is a vector representing the three-dimensional position of the pixel (i, j), and is calculated from the correction distance data 804 as follows. First, based on the pixel value of the correction distance data 804, a virtual distance value from the imaging device 101 to the subject position corresponding to each pixel is calculated. At this time, it is assumed that a pixel having a larger pixel value in the correction distance data 804 has a smaller distance from the imaging device 101. Subsequently, the lighting unit 305 determines the three-dimensional position P of the pixel (i, j) based on the virtual distance value corresponding to each pixel, the angle of view of the imaging device 101, the image size of the color image data 501, and the like. Is calculated. W is a function that returns a larger value as the distance from the position P of the pixel (i, j) to the position Q of the light source increases. ρ represents an angle formed by a vector from Q to P (i, j) and an illumination posture U. K is a function that increases as ρ decreases. N (i, j) is a normal vector corresponding to the pixel (i, j), and V (i, j) is a unit vector representing a direction from Q to P (i, j). By generating a corrected image as in the present embodiment, it is possible to correct the brightness according to the position of the illumination and the shape of the subject. As described above, the lighting process for adding pixel values according to the distance from the virtual light source is performed. With the above processing, it is possible to perform correction so that a pixel closer to the virtual light source and having a smaller angle formed by a vector from the virtual light source toward the pixel (i, j) and a normal vector becomes brighter. Thereby, as shown in FIG. 14, it is possible to obtain a corrected image 1401 as if the subject was illuminated by virtual illumination.

  In step S1203, the lighting unit 305 displays the corrected image data on which the pixel value is corrected on the display unit 106, and ends the process. The user views the corrected image data displayed on the display unit 106 here, and inputs an illumination parameter change instruction and an image output instruction.

  According to the above processing, the lighting process can be performed using the face normal model deformed according to the subject, so that an image with a natural shadow according to the posture and expression of the subject can be obtained.

  In the present embodiment, the developing unit 301 functions as an image acquisition unit that acquires image data representing an image including a subject. Further, the distance correction unit 302 functions as a distance acquisition unit that acquires distance information indicating the distance from the imaging device that captured the subject to the subject to each pixel of the image data. The distance correction unit 302 also functions as a correction unit that corrects the distance information. The face detection unit 303 functions as a detection unit that detects a face in the image represented by the image data. The normal correction unit 304 functions as a deforming unit that generates corrected face data by deforming the face model data based on the face detected by the detecting unit. The normal line correction unit 304 also functions as normal line acquisition means for acquiring normal line information indicating the normal direction on the surface of the subject for each pixel of the image data based on the distance information. The lighting unit 305 functions as a processing unit that performs a lighting process for correcting a pixel value of the image data based on the corrected face data, the distance information, and the position of the light source. The ROM 203 functions as a holding unit that holds face model data representing a three-dimensional shape of a predetermined face.

[Embodiment 2]
In the first exemplary embodiment, the pixel value of the color image data is corrected regardless of the pixel value of the subject. In the second embodiment, a method for controlling the correction amount based on the luminance value of the subject will be described. By controlling the correction amount based on the luminance value of the subject, it is possible to suppress overexposure that occurs when an area with a high luminance value is corrected in advance, and noise increase that occurs when a dark part is corrected.

  Since the configuration of the imaging apparatus 101 according to the present embodiment and the basic processing flow are the same as those in the first embodiment, description thereof is omitted. The second embodiment is different from the first embodiment in that the correction coefficient t shown in Expression 4 is determined by the luminance value of each pixel in the lighting processing performed by the lighting processing unit 305.

  An example of a method for determining the correction coefficient t in the present embodiment will be described with reference to FIG. FIG. 15A shows an example in which the correction coefficient t is determined based on preset threshold values th1 and th2. In this example, the pixel brightness value Y is corrected so that t = 1 in a section where 0 ≦ Y <th1, t decreases monotonously in a section where th1 ≦ Y <th2, and t = 0 in a section where th2 ≦ Y. A coefficient t is determined. When the correction coefficient t is determined in this way, the degree of correction by the virtual light source can be reduced as the luminance value of the pixel increases. Therefore, an effect of suppressing overexposure of pixels having a high luminance value due to the influence of the virtual light source can be obtained. In FIG. 15 (a), t is linearly decreased so that t becomes a linear function of Y in the section of th1 ≦ Y <th2, but the method of decreasing is not limited to this. When t is a linear function of Y, the corrected image I ′ is expressed as a quadratic function of the pixel values of the color image data I. In this case, the pixel value of the corrected image becomes maximum in the section of th1 ≦ Y <th2, and gradation inversion may occur. As a means for suppressing this, a decrease in the interval of th1 ≦ Y <th2 may be expressed using a quadratic curve, a trigonometric function, or the like. By doing so, the occurrence of gradation inversion in the corrected image can be suppressed. Alternatively, the correction coefficient t can be determined as shown in FIG. FIG. 15B shows an example of correction coefficients determined by preset threshold values th1, th2, and th3. In this example, t is monotonically increasing when the luminance value Y is 0 ≦ Y <th3, t = 1 when t3 ≦ Y <th1, t is monotonically decreasing when th1 ≦ Y <th2, and th2 ≦ Y. In the interval, the correction coefficient t is determined so that t = 0. By setting the correction coefficient t as shown in FIG. 15B, it is possible to suppress the dark portion noise having a small luminance value in addition to the overexposure from being emphasized by the lighting processing. Further, by expressing the increase in the brightness value Y in the interval of 0 ≦ Y <th3 and the decrease in the interval of th1 ≦ Y <th2 using a quadratic curve, a trigonometric function, or the like, FIG. As in the case of, the occurrence of gradation inversion can be suppressed.

  As described above, according to the processing of the present embodiment, it is possible to suppress overexposure that occurs when a region having a high luminance value is corrected in advance, and noise increase that occurs when a dark portion is corrected.

[Embodiment 3]
In the first embodiment and the second embodiment, the example in which the lighting process is performed to brighten the subject that appears dark by applying a virtual light source to the scene has been described. In the third embodiment, a method of enhancing the stereoscopic effect of a subject by adding a shadow to the subject that appears flat due to the effect of light emission from a strobe will be described.

  Since the configuration of the imaging apparatus 101 according to the present embodiment and the basic processing flow are the same as those in the first embodiment, description thereof is omitted. The third embodiment is different from the first embodiment in that the pixel value correction process performed in step S1202 is different. Hereinafter, the process performed in step S1202 of the present embodiment will be described. In step S1202 of the present embodiment, unlike in the first embodiment, pixel values are corrected based on the following equation (5).

  The difference from Expression (3) is that the pixel value is corrected so that the pixel value of the color image data becomes smaller according to k′m. That is, what is performed in the present embodiment is a lighting process that subtracts pixel values according to the distance from the virtual light source. k'm is determined based on the brightness value α, the position Q, the posture U, the distance value corresponding to the pixel (x, y), and the normal vector V. For example, it can obtain | require like Formula (6).

  The difference from equation (4) is mainly the influence of the angle formed by the normal vectors N (i, j) and V (i, j). In equation (4), the value of k increases as the normal vector N faces the virtual light source direction. In equation (6), the value of k increases as the normal vector N faces the virtual light source direction. Becomes smaller. That is, according to Expression (6), a shadow that is closer to the virtual illumination and whose normal vector N does not face the virtual light source direction can be given a stronger shadow. As a result, as in the corrected image 1601 shown in FIG. 16, it is possible to add a shadow only to the cheek or nose of the face based on the normal image data.

  According to the above processing, it is possible to perform correction to add a shadow so that a subject that appears flat due to the influence of light emission from a strobe or the like has a three-dimensional effect.

[Embodiment 4]
In the above-described embodiment, the relighting process for applying a virtual light source to the scene and the relighting process for adding a shadow to the image have been described. In the fourth embodiment, a method of switching the above two processes based on the shooting conditions will be described. FIG. 17 is a flowchart illustrating an operation procedure of the image processing unit 209 according to the fourth embodiment. Compared with the first embodiment, step S1701 and step S1702 are newly added.

  In step S1701, the lighting unit 305 acquires information regarding the actual light source. Here, the actual light source is a light source that actually exists in a space for imaging a subject. In this embodiment, the control unit 206 determines the presence or absence of strobe light emission based on a user's instruction to use the strobe or an input signal from the strobe 104, and informs the lighting unit 305 whether or not the strobe was used when image data was captured. I want to output. If it is determined that the strobe is used when the image data is captured, it is acquired that the strobe disposed at the predetermined position Q ′ is emitting light. Note that the method of acquiring information about the actual light source is not limited to this. For example, the average luminance of the pixels in the face area of the selected subject is obtained. You may judge.

  In step S1702, the lighting unit 305 sets a lighting processing mode based on the real light source information acquired in step S1701. In this step, if it is determined in step S1701 that the strobe light is not emitted at the time of imaging, the lighting mode for providing the virtual light source shown in the first embodiment is set. If it is determined in step S1701 that the strobe is emitting light at the time of imaging, the lighting mode for applying a shadow shown in the third embodiment is set.

  In step S411, the lighting unit 305 performs a lighting process corresponding to the lighting mode set in step S1702 on the color image data to generate corrected image data.

  The above is the flow of processing in this embodiment. According to the above processing, an appropriate lighting process can be selected according to the state of the light source when the subject is imaged.

  Note that the processing of the present embodiment is not limited to the above. For example, the strobe light position Q ′ may be acquired as the actual light source information in step S1701, and the strobe light position Q ′ may be input as the initial value of the illumination parameter in the lighting process in step S411. Further, in the color image data, it is assumed that an actual light source other than the strobe is present in an area where the luminance is larger than a predetermined threshold, and when the detected actual light source is present at a position closer to the imaging device 101 than the subject, The lighting mode may be a shadowing mode. Further, the position of the actual light source may be acquired from the color image data and input as the initial value of the illumination parameter.

<Other embodiments>
The embodiment of the present invention is not limited to the embodiment described above. For example, the lighting process may be performed using the subject distance information directly without using the normal image data in the lighting process. In this case, since it is necessary to use a calculation formula different from the above formula, the processing becomes complicated, but the same effect as the present invention can be obtained. At that time, a 3D model of a predetermined face may be held instead of the face normal model. That is, in the embodiment of the present invention, information indicating the three-dimensional shape of the subject can be widely used. Alternatively, a 3D model of a predetermined face may be held instead of the face normal model, and normal information may be acquired based on the deformed 3D model.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

301 Development Unit 302 Distance Correction Unit 303 Face Detection Unit 304 Normal Correction Unit 305 Writing Unit

Claims (27)

An image processing apparatus that generates a corrected image in which lighting on the subject is changed in a captured image obtained by imaging the subject,
Image acquisition means for acquiring the captured image;
The distance from the imaging device imaging the subject to the subject, a distance obtaining means for obtaining distance information indicating for each pixel of the captured image,
Setting means for setting illumination parameters relating to a virtual light source for generating the corrected image; holding means for holding face model data representing a three-dimensional shape of a predetermined face;
Detecting means for detecting a face area of the subject in the captured image as a face area;
Deforming means for generating corrected face data by deforming the face model data based on the face area detected by the detecting means;
Processing means for performing a lighting process for correcting a pixel value of each pixel in the captured image using a correction amount according to the correction face data, the distance information, and the illumination parameter;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the deformation unit outputs data in which normal line information is stored for each pixel in the face area of the captured image as the corrected face data.   The image processing apparatus according to claim 1, wherein the processing unit uses a correction amount corresponding to the unevenness of the face in the face area.   The said processing means uses the said correction amount which does not depend on the unevenness | corrugation of the to-be-photographed object in the area | regions of the to-be-photographed object other than the said face area in the said captured image. Image processing apparatus. The processing means, in the background area other than the area of the subject in the captured image according to claim 1 or 4 wherein the value smaller than the correction amount, which is characterized by using as said correction amount to be used in the region of the subject The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 1, wherein the processing unit calculates the correction amount for each pixel in the captured image.   The processing means calculates, for the target pixel in the captured image, a distance of the target pixel in the distance information, information indicating a three-dimensional shape of the target pixel in the corrected face data, and a correction degree according to the illumination parameter. The image processing apparatus according to claim 1, wherein the correction amount is calculated by adding the correction degree to a pixel value of the target pixel. The illumination parameter includes information indicating a position of the virtual light source,
The image processing apparatus according to claim 1, wherein the processing unit calculates the correction amount so that a pixel closer to the virtual light source has a larger correction amount. A correction means for correcting the distance information acquired by the distance acquisition means;
The image processing apparatus according to claim 7, wherein the processing unit acquires the corrected distance information from the correction unit, and calculates the correction degree based on the acquired distance information. The correction means includes
In the captured image, obtain information indicating the pixel position designated by the user,
Based on the pixel position, a subject whose distance from the subject instructed by the user is included in a predetermined range and a subject whose distance from the subject instructed by the user is not included in the predetermined range are determined.
The distance information corresponding to the subject whose distance from the subject instructed by the user is included in the predetermined range is corrected to the first distance value, and the distance from the subject instructed by the user is included in the predetermined range. The image processing apparatus according to claim 9, wherein distance information corresponding to a non-subject is corrected to a second distance value that is larger than the first distance value. The correction unit further performs a smoothing process based on the pixel value of the captured image on the region corrected to the first distance value of the distance information corrected by the correction unit. the image processing apparatus according to claim 10.   The image processing apparatus according to claim 11, wherein the smoothing process is a process using a joint bilateral filter.   The face model data is normal image data in which information indicating a normal direction of the predetermined face surface is stored in each pixel of an image indicating the predetermined face. The image processing apparatus described.   The deformation means is deformed by applying affine transformation to the face model data, and the corrected face data is generated by correcting normal information of each pixel in the deformed face model data. The image processing apparatus according to claim 13, characterized in that: The face model data has a normal direction of the predetermined face surface as a component in directions of three coordinate axes orthogonal to each other in each pixel of an image showing the predetermined face,
The image processing apparatus according to claim 14, wherein the deformation unit causes the joint bilateral filter to act on each component in the direction of the coordinate axis. The holding means further holds information indicating a position of at least one organ in the predetermined face;
The detection means acquires the position of the at least one organ included in the face area detected in the captured image,
The deforming means deforms the face model data based on the position of the at least one organ in the predetermined face and the position of the at least one organ acquired by the detecting means. Item 16. The image processing device according to any one of Items 1 to 15.   The deforming means deforms the face model data so that the position of the at least one organ in the predetermined face matches the position of the at least one organ acquired by the detecting means. The image processing apparatus according to claim 16.   The image processing apparatus according to claim 16, wherein the organ includes an eye.   The image processing apparatus according to claim 16, wherein the organ includes a mouth.   The image processing apparatus according to claim 16, wherein the organ includes a nose.   The image processing apparatus according to claim 1, wherein the processing unit adds the correction amount to a pixel value of each pixel in the captured image.   The image processing apparatus according to claim 1, wherein the processing unit subtracts the correction amount from a pixel value of each pixel in the captured image. The processing means acquires information indicating the presence or absence of strobe emission when the captured image is captured,
If strobe light is emitted when the captured image is captured, the correction amount is subtracted from the pixel value of each pixel in the captured image,
The correction amount is added to a pixel value of each pixel in the captured image as the lighting process when the flash is not emitted when the captured image is captured. The image processing apparatus according to any one of claims.   The said processing means determines the correction amount of the pixel value of each pixel in the said captured image based on the pixel value of each pixel in the said captured image, The one of Claim 1 thru | or 23 characterized by the above-mentioned. Image processing apparatus.   The image processing apparatus according to claim 7, wherein in the captured image, the processing unit reduces the degree of correction as the pixel value increases. An image processing method for generating a corrected image in which lighting for a subject is changed in a captured image obtained by capturing a subject using face model data representing a three-dimensional shape of a predetermined face held in advance. And
Obtaining the captured image;
The distance from the imaging device imaging the subject to the subject, a distance obtaining step of obtaining distance information indicating for each pixel of the captured image,
Set illumination parameters for a virtual light source for generating the corrected image,
Detecting the face area of the subject in the captured image as a face area;
Deforming the face model data based on the detected face area to generate corrected face data; and
An image processing method comprising: performing lighting processing for correcting a pixel value of each pixel in the captured image using a correction amount according to the corrected face data, the distance information, and the illumination parameter.   A program that causes a computer to function as the image processing apparatus according to any one of claims 1 to 25.

Images