JP2021145385A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To obtain an image provided with a shadow according to a posture and facial expression of a subject.SOLUTION: The image processing apparatus includes: acquisition means for acquiring a color image obtained by imaging a subject; specifying means for specifying a face area of the subject as a face area in the color image; and generation means for generating a corrected image formed by modifying brightness of at least a part of the face area in the color image.SELECTED DRAWING: Figure 4

Description

The present invention relates to a technique for imparting shading to image data.

When a subject is photographed using an imaging device, the image obtained varies greatly depending on how the light hits the subject. For example, when shooting against the sun, all or part of the subject becomes a shadow and appears dark. Further, when the subject is irradiated with light using a strobe for shooting, the shadow of the subject may be skipped due to the influence of the light, and the subject may appear flat. As a method of correcting these images, there is known a method of correcting a dark part of a subject as if light is applied from a specific direction in a pseudo manner (Patent Document 1). In the technique described in Patent Document 1, the influence of a virtual light source is expressed by a Gaussian distribution. At this time, by biasing the Gaussian distribution of the brightness correction according to the lighting direction set separately, it is possible to obtain an image as if the illumination was applied from a virtually desired direction.

In addition, the lighting of the subject is changed by performing CG rendering processing under predetermined virtual lighting conditions using a 3D model corresponding to the subject and replacing the rendered CG image with the subject image in the image obtained by shooting. A method for rendering is also known (Patent Document 2). In the technique described in Patent Document 2, a 3D model to be replaced with the subject is determined from the 3D models prepared in advance.

Japanese Patent No. 5281878 Japanese Patent No. 5088220

When another image is generated from the captured image as if it was captured under the lighting conditions desired by the photographer, it is preferable to obtain an image with shading according to the posture and facial expression of the subject. However, the techniques of Patent Documents 1 and 2 have the following problems.

When the influence of a virtual light source is expressed by a Gaussian distribution as in the technique described in Patent Document 1, it is difficult to give a natural shadow to a subject having a complicated shape such as a human face. Further, when the subject image in the image is replaced with the result of rendering using the 3D model of the subject as in the technique described in Patent Document 2, the posture or facial expression of the subject after the replacement becomes the posture or facial expression of the 3D model. It will be replaced and the result will be unnatural. Therefore, an object of the present invention is to obtain an image in which a shadow is added according to the posture and facial expression of the subject.

In order to solve the above problems, the image processing apparatus according to the present invention specifies an acquisition means for acquiring a color image obtained by photographing a subject and a region of the face of the subject as a face region in the color image. It is characterized by having a specific means for generating a corrected image in which the brightness of at least a part of the face region in the color image is changed.

According to the present invention, it is possible to obtain an image in which a shadow is added according to the posture and facial expression of the subject.

The figure which shows the appearance of the image pickup apparatus which concerns on Embodiment 1 of this invention. The figure which shows the internal structure of the image pickup apparatus which concerns on Embodiment 1 of this invention. The block diagram which shows the structure of the image processing part which concerns on Embodiment 1 of this invention. The flowchart which shows the flow of the process which concerns on Embodiment 1 of this invention. The figure which shows the example of the image data which concerns on Embodiment 1 of this invention. The figure which shows the face information which concerns on Embodiment 1 of this invention. The flowchart which shows the flow of the distance correction processing which concerns on Embodiment 1 of this invention. The figure which shows the outline of the distance correction processing which concerns on Embodiment 1 of this invention. The flowchart which shows the flow of the normal correction processing which concerns on Embodiment 1 of this invention. The figure which shows the outline of the normal correction processing which concerns on Embodiment 1 of this invention. The figure which shows the outline of the normal smoothing process which concerns on Embodiment 1 of this invention. The flowchart which shows the flow of the lighting process which concerns on Embodiment 1 of this invention. The figure which shows the outline of the lighting process which concerns on Embodiment 1 of this invention. The figure which shows the effect of the lighting process which concerns on Embodiment 1 of this invention. The figure which shows the example of the correction coefficient which concerns on Embodiment 2 of this invention. The figure which shows the effect of the lighting process which concerns on Embodiment 3 of this invention. The flowchart which shows the flow of the process which concerns on Embodiment 4 of this invention.

[Embodiment 1]
<Appearance of imaging device>
1A and 1B are views showing the appearance of the image pickup apparatus according to the present embodiment, FIG. 1A shows the appearance of the front surface of the image pickup apparatus, and FIG. 1B shows the appearance of the back surface of the image pickup apparatus. The image pickup apparatus 101 includes an optical unit 102, an image pickup button 103, a strobe 104, a distance acquisition unit 105, a display unit 106, and an operation button 107.

The optical unit 102 is a lens barrel composed of a zoom lens, a focus lens, a blur correction lens, an aperture, and a shutter, and collects light information of a subject. The image pickup button 103 is a button for the user instructing the image pickup apparatus 101 to start imaging. The strobe 104 is an illumination that can emit light at the start of imaging according to a user instruction. The distance acquisition unit 105 is a distance acquisition module that acquires distance image data of a subject in response to an imaging instruction. The distance image data means image data in which the subject distance corresponding to the pixel is stored as the pixel value of each pixel of the image. The distance acquisition unit 105 includes an infrared light emitting unit that emits infrared light and a light receiving unit that receives infrared light reflected by the subject, and takes a time until the emitted infrared light is reflected by the subject and received. Based on this, the distance value from the image pickup device to the subject is calculated. Then, the position information of the subject is calculated based on the calculated distance value and the distance imaging information including the number of sensor pixels of the light receiving unit, the angle of view, and the like, and the distance image data is generated. The method of acquiring the distance image data is not limited to this. For example, instead of the distance acquisition unit 105, an optical system similar to the optical unit 102 is provided, and distance image data is acquired by performing triangulation based on the parallax between the image data captured from two different viewpoints. It may be.

The display unit 106 is a display such as a liquid crystal display that displays image data processed by the image pickup apparatus 101, various other data, and the like. Since the image pickup apparatus 101 is not provided with an optical viewfinder in the present embodiment, the framing operation (confirmation of focus and composition) is performed using the display unit 106. That is, since the image is taken while checking the live view image on the display unit 106, it can be said that the display unit 106 functions as an electronic viewfinder while performing framing and focusing operations. On the display unit 106, a live view display for displaying the imaging range in real time is performed, and a camera setting menu is displayed.

The operation button 107 is a button for the user to instruct the image pickup apparatus 101 of an operation for switching the operation mode of the image pickup apparatus 101, various parameters at the time of imaging, and the like. In the present embodiment, as one of the operation modes, a lighting correction processing mode for correcting the lighting condition in the captured image after imaging is included. The user can switch to the lighting correction processing mode using the operation button 107 or the image pickup button 103, set the lighting parameters of the virtual lighting used for the lighting correction, select the subject for adjusting the lighting condition, and the like. can. Further, when outputting the corrected image data, the user can also give an instruction as to whether or not to output the distance image data. The display unit 106 may have a touch screen function, and in that case, a user instruction using the touch screen can be treated as an input of the operation button 107.

<Internal configuration of imaging device>
FIG. 2 is a block diagram showing an internal configuration of the image pickup apparatus 101 according to the present embodiment.

The CPU 202 is involved in all the processes of each configuration, reads and interprets the instructions stored in the ROM (Read Only Memory) 203 and the RAM (Random Access Memory) 204 in order, and executes the processes according to the result. The system bus 212 is a bus for transmitting and receiving data. In the present embodiment, it is assumed that the face normal model corresponding to the human face is stored in the ROM 203. The face normal model shows normal image data in which the normal vector of the face surface corresponding to a face having a predetermined shape is stored in pixel values, and the positions of human organs such as eyes and mouth in the normal image data. Includes organ location information.

The control unit 206 is a control circuit that receives user instructions from the image pickup button 103 and the operation button 107 and controls imaging, switching of the lighting correction processing mode, selection of a subject area, setting of lighting parameters, and the like. The optical system control unit 205 is a control circuit that controls the optical unit 102 instructed by the CPU 202, such as focusing, opening a shutter, and adjusting the aperture.

The color image sensor unit 201 is an image sensor that converts the light information collected by the optical unit 102 into a current value. The color image pickup element unit 201 is provided with a color filter having a predetermined arrangement such as a Bayer arrangement, and color information of a subject is acquired from the light focused by the optical unit 102.

The A / D conversion unit 208 is a processing circuit that converts the color information of the subject detected by the color image sensor unit 201 into digital signal values and converts them into RAW image data. In this embodiment, it is assumed that the distance image data and the RAW image data captured at the same time can be acquired.

The image processing unit 209 develops the RAW image data acquired by the A / D conversion unit 208 to generate color image data. Further, the image processing unit 209 performs various image processing such as generating corrected image data in which lighting correction is performed on the color image data by using the color image data and the distance image data. The internal structure of the image processing unit 209 will be described in detail later.

Further, the character generation unit 207 is a processing circuit that generates characters, graphics, and the like. The characters and graphics generated by the character generation unit 207 are superimposed on the image data, the corrected image data, and the like and displayed on the display unit 106.

The encoder unit 210 performs a process of converting various image data including the color image data processed by the image processing unit 209 and the corrected image data generated by the lighting correction process into a file format such as JPEG.

The media I / F211 is an interface for transmitting and receiving image data to and from a PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.). As the media I / F211 for example, USB (Universal Serial Bus) or the like is used.

<Internal configuration of image processing unit>
FIG. 3 is a block diagram showing a functional configuration of the image processing unit 209 in the present embodiment. The development processing unit 301 performs white balance processing, demosaic processing, noise reduction processing, color conversion processing, edge enhancement processing, gamma processing, and the like on the RAW image data acquired from the A / D conversion unit 208, and obtains the color image data. Generate. The generated color image data can be output to the display unit 106 for display or stored in a storage device such as a RAM 204 or a PC / media 213. In this embodiment, the development processing unit 301 generates color image data without performing gamma processing and outputs the color image data to the lighting unit 305.

The distance correction unit 302 generates correction distance data corresponding to the subject selected from the distance image data based on the color image data, face information, and the subject position selected by the user. In the present embodiment, it is assumed that the correction distance data mainly stores a person corresponding to the selected subject position and a distance value corresponding to other backgrounds.

The face detection unit 303 acquires the face information of the subject from the color image data acquired from the development processing unit 301. The face information of the subject includes at least information on a face region indicating an area occupied by the subject's face in the color image data and an organ position indicating a position in the color image data such as eyes and mouth included in the face.

The normal correction unit 304 corrects the face normal model stored in the ROM 203 based on the face information acquired from the face detection unit 303 and the color image data acquired from the development unit 301.

The lighting processing unit 305 refers to the color image data based on the correction distance data acquired from the distance correction unit 302, the correction normal data acquired from the normal correction unit 304, and the illumination parameters acquired from the control unit 206. And perform the lighting process. The corrected image data generated by the lighting process can be output and stored in a storage device such as a RAM 204 or a PC / media 213, or can be output and displayed on the display unit 106.

<Processing flow of image processing unit>
FIG. 4 is a flowchart showing an operation procedure of the image processing unit 209 in the image pickup apparatus of the present embodiment. In the present embodiment, the image processing unit 209 uses the face information acquired from the color image data and the subject position P0 acquired based on the user's instruction to obtain the correction distance data corresponding to the selected subject from the distance image data. Generate. Then, based on the face information of the subject and the face normal model held in advance, normal image data matching the face of the subject is generated. After that, based on the lighting parameters set by the user operation, the correction distance data, and the generated normal image data, a lighting process of adding a virtual light source to the color image data is performed to generate the correction image data. The details of the operation procedure of the image processing unit 209 will be described below.

In step S401, the development processing unit 301 performs development processing such as demosaic processing on the RAW image data acquired from the A / D conversion unit 208 to generate color image data. The color image data in this embodiment will be described with reference to FIG. 5 (a). It is assumed that RGB values are stored as pixel values in the pixels (i, j) of the color image data I501, and they are represented as Ir (i, j), Ig (i, j), and Ib (i, j), respectively. And. The method for acquiring color image data is not limited to this. For example, the RAW image data stored in the RAM 204 or the PC / media 213 may be acquired and the development processing unit 301 may generate the color image data. Alternatively, the color image data that has already been developed may be acquired from the RAM 204 or the PC / media 213. Then, in step S402, the developing unit 301 outputs the color image data acquired in step S401 to the display unit 106. The user determines whether or not to perform the lighting correction process based on the display on the display unit 106.

In step S403, the control unit 206 determines whether or not an instruction to perform the lighting correction process is input according to the input from the operation unit 107. If the instruction to perform the lighting correction process is not input, the process proceeds to step S404. When an instruction to perform the lighting correction process is input, the control unit 206 outputs a signal indicating that the lighting correction is performed to the developing unit 301 and the distance correction unit 302, and proceeds to step S406.

In step S404, the control unit 206 determines whether or not an image output instruction has been input by the user. If it is determined that the image output instruction has been input by the user, the process proceeds to step S405. If it is not determined that the image output instruction has been input by the user, the process returns to step S403.

In step S405, the control unit 206 outputs an image output instruction to the developing unit 301, and the developing unit 301 outputs the color image data to the PC / media 213 to end the process.

In step S406, the distance correction unit 302 acquires the distance image data from the distance acquisition unit 105. The distance image data in this embodiment will be described with reference to FIG. 5 (b). It is assumed that the pixel (i, j) of the distance image data D502 stores the distance value D (i, j) from the image pickup apparatus to the subject as a pixel value. The method of acquiring the distance image data is not limited to this. For example, the distance image data stored in the RAM 204 or the PC / media 213 may be acquired.

In step S407, the developing unit 301 outputs the color image data to the face detecting unit 303, and the face detecting unit 303 acquires the face information of the subject from the input color image data. The face information in this embodiment will be described with reference to FIG. The face information in the present embodiment includes information indicating the face area 601 and the organ position 602. The face region represents a set of pixels in a region including a face in the color image data 501. The organ position 602 represents the coordinates corresponding to the eyes and mouth in the facial area. Existing algorithms can be applied to the method of detecting the facial region and organ position. Examples include an algorithm using template matching and an algorithm using Haar-Like features. In this embodiment, the face region / organ position is detected by template matching. First, the skin color region is extracted as a face candidate region by performing threshold processing on the color image data. That is, pixels whose pixel values fall within the range of pixel values determined based on various skin colors are extracted as face candidate regions. Then, matching processing is performed on the face candidate region using face image templates of various sizes, and the likelihood as the face region is calculated. Finally, a region whose calculated likelihood is equal to or higher than a predetermined threshold value is extracted as a face region. Further, the face detection unit 303 performs the same template matching on the extracted face area using the eye and mouth image templates, and extracts the coordinates corresponding to the eyes and mouth. By the above processing, the face area 601 and the organ position 602 are acquired. The face detection unit 303 outputs the acquired face information to the normal correction unit 304. In addition to the eyes and mouth, other organs such as the nose and ears may be extracted as the organs to be detected.

In step S408, the distance correction unit 302 determines the position of the subject designated by the user. In the present embodiment, the user uses the touch panel provided on the display unit 106 and the operation buttons 107 to specify the position of the subject to be subjected to the lighting correction processing. The distance correction unit 302 acquires the subject selection position P0'input by the user operation from the control unit 206. Then, the designated subject position P0 in the color image data is calculated based on the acquired subject selection position P0'. In the present embodiment, the color image data is displayed on the display unit 106 having the touch screen function, the operation of the user touching the subject on the display screen is accepted, and the distance correction unit 302 sets the position touched by the user as the subject selection position. Obtained from the control unit 206 as P0'. At this time, the subject selection position P0'corresponds to the pixel position of the display unit 106. The distance correction unit 302 calculates the subject position P0 by converting the pixel position on the display unit 106 into the pixel position of the color image data.

In step S409, the distance correction unit 302 generates correction distance data from the distance image data acquired in step S406 by using the subject position P0 acquired in step S408 and the color image data acquired from the developing unit 301. The details of the correction distance data generation process will be described later. The distance correction unit 302 outputs the generated correction distance data to the lighting unit 305.

In step S410, a correction method that is normal image data that matches the face of the subject based on the face information acquired by the normal correction unit 304 from the face detection unit 303 and the color image data input from the development unit 301. Generate line data. The details of the correction normal data generation process will be described later. The normal correction unit 304 outputs the generated correction normal data to the lighting unit 305.

In step S411, the lighting unit 305 performs lighting processing such as adding a virtual light source to the color image data based on the input correction distance data and the correction normal data to generate the correction image data. The details of the lighting process will be described later.

In step S412, the lighting unit 305 determines whether or not a change in the setting of the lighting parameter used for the lighting process has been input from the control unit 206. If it is determined that the lighting parameter setting has been changed, the process returns to step S411 and the lighting process is performed again. If it is determined that the lighting parameter settings have not been changed, the process proceeds to step S413.

In step S413, the lighting unit 305 determines whether or not an image output instruction has been input from the control unit 206. If it is determined that the image output instruction has been input, the process proceeds to step S414. If it is determined that the image output instruction has not been input, the process returns to step S412. In step S414, the lighting processing unit 305 outputs the generated corrected image data to the PC / media 213 and ends the processing. The above is the flow of processing performed by the image processing unit 209 of the present embodiment. According to the above processing, since the lighting processing can be performed using the face normal model deformed according to the subject, it is possible to obtain an image with natural shading according to the posture and facial expression of the subject. Hereinafter, the details of the processing performed in each component of the image processing unit 209 will be described.

<Correction distance data generation process>
Here, the correction distance data generation process performed by the distance correction unit 302 in step S409 will be described with reference to the flowchart shown in FIG. 7. In step S701, the distance correction unit 302 extracts the subject candidate region based on the face information, the subject position P0, and the distance image data. The process of this step will be described with reference to FIGS. 8A and 8B. First, the distance correction unit 302 selects the face area 601 closest to the subject position P0 from the face areas indicated by the face information. Then, the distance value of each pixel in the selected face region is acquired from the distance image data, and the average value thereof is calculated as the distance value of the face region. After that, the distance correction unit 302 generates a binary image 801 divided into pixels in which the difference between the distance value in the face region and the distance value is equal to or less than a predetermined threshold value and other pixels. That is, the process performed here is a process of discriminating between a subject whose distance from the selected subject is within a predetermined range and a subject other than that. Here, in the binary image 801, the pixel in which the difference between the distance value of the face region and the distance value is equal to or less than the threshold value is defined as the subject candidate region 802. The determination of the subject candidate area performed here is not limited to the above method, and an area in which the difference in the distance value from the selected subject position is within a predetermined threshold value may be determined as the subject candidate area.

In step S702, the distance correction unit 302 performs a small component removal process or a hole filling process on the binary image 801 to remove a small connected component included in the subject candidate region or perform a shaping process to fill the hole. As the small component removal process / fill-in-the-blank process, a method using a morphology calculation or a method using a labeling process can be applied. Here, a method using morphology calculation is used. The distance correction unit 302 performs an opening process on the subject candidate region included in the binary image 801 as a small component removal process. Then, as the subsequent fill-in-the-blank process, a closing process is performed on the subject candidate area. FIG. 8C shows an example of the binary image 803 obtained by this step.

In step S703, the distance correction unit 302 performs smoothing processing on the binary image 803 subjected to the shaping process in step S702 to generate multi-value correction distance data 804 (FIG. 8D). For example, in the binary image 803, a smoothing process is performed on an image in which the pixel value of the pixel included in the subject candidate area 802 is 255 and the pixel value of the other pixels is 0, and the distance of 8 bits per pixel. The correction distance data 804 having information is generated. At this time, it is assumed that the larger the pixel value, the smaller the distance to the subject.

As the smoothing process, a Gaussian filter, a joint bilateral filter that performs smoothing while referring to the pixel values of color image data, and the like can be applied. In this embodiment, the joint bilateral filter represented by the following equation (1) is used.

s is the pixel to be processed, Ω is the neighborhood region of s, p is the pixel included in Ω, I is the image data for smoothing, R is the reference image data, and f is the weight based on the distance between p and s. g represents a weight based on the pixel value. f is set so that the weight decreases as the distance between s and p increases. g is set so that the larger the difference between the pixel values of the pixel p and the pixel s of the reference image, the smaller the weight. In the formula (1), Y represents the brightness difference between the pixel values of the pixel p and the pixel s. In step S703, smoothing processing is performed using the correction distance data 803 as I and the color image data as R. By using a joint bilateral filter for the binary image 803 while referring to the color image data, it is possible to perform the smoothing process using only the pixels having similar pixel values in the color image data. As a result, the contour of the subject area 802 can be smoothed while matching the contour of the subject in the color image data. The method of smoothing processing is not limited to this. For example, as a method of setting f, equal weights may be given in the neighborhood region. Further, as a method of setting g, a weight may be used based on a color difference instead of a luminance value. Alternatively, if the pixel value is within a certain value, the weight may be made constant.

Further, by acquiring the correction distance data 804 as a multi-valued image, it is possible to reduce the discomfort of the subject contour portion during the lighting process performed in step S409. By the above processing, the distance correction unit 302 is mainly divided into a subject in the foreground and a background other than the subject, and can acquire the correction distance data 804 in which the distance values corresponding to the respective subjects are stored. The filter processing performed here does not have to be a joint bilateral filter, and any filter processing based on the pixel value of the color image data may be used.

<Normal image data generation processing>
Here, the correction normal data generation process performed by the normal generation unit 304 in step S410 will be described. The correction normal data generation process in the present embodiment is a process for correcting the face normal model stored in the ROM 203 or the PC / media 213 based on the color image data. Hereinafter, the details of the correction normal data generation process will be described with reference to the flowchart shown in FIG.

In step S901, the deformation parameter when the face normal model is deformed according to the color image data is calculated. An example of the face normal information of this embodiment is shown in FIG. 10 (a). The face normal model includes face normal image data 1001 and corresponding organ position information 1002. The face normal image data 1001 stores the normal vector of the face orientation (Nx (i, j), Ny (i, j), Nz (i, j)) as a pixel value in the pixel N (i, j). This is the image data. Nx (i, j), Ny (i, j), and Nz (i, j) are the x-axis and y, which are three coordinate axes of the normal vector stored in the pixel (i, j), which are orthogonal to each other. It is a component in the axial and z-axis directions. Further, all the normal vectors included in the face normal image data 1001 are unit vectors. For the pixels corresponding to the face area, the vector in the direction perpendicular to the face surface is stored as a normal vector, and for the pixels corresponding to the area other than the face, the vector in the direction opposite to the optical axis of the image pickup device is the normal vector. It is assumed that it is stored as. In the present embodiment, the z-axis is in the direction opposite to the optical axis of the image pickup apparatus, and (0, 0, 1) is stored as a normal vector in the pixels corresponding to the region other than the face. The organ position information 1002 indicates the coordinate values of the right eye, the left eye, and the mouth in the face normal image data 1001.

In this step, the normal correction unit 304 uses the organ position 1002 corresponding to the face normal model and the organ position 602 included in the face information of the color image data to obtain the color image data 501 and the face normal image data 1001. Associate the coordinates of the right eye, left eye, and mouth of. Then, the normal correction unit 304 calculates a deformation parameter for adjusting the organ position 1002 of the face normal image data 1001 to the organ position 602. As the deformation parameter, the affine transformation coefficient for use in the affine transformation is calculated. As a method for calculating the affine transformation coefficient, the least squares method or the like can be used. That is, the affine transformation coefficient that minimizes the circumstance of the error from the organ position 602 when the organ position 1002 is affine-transformed is determined as the conversion parameter here. In the present embodiment, the face normal image data 1001 has components in the x-axis, y-axis, and z-axis directions of the normal vector as pixel values. For example, these are included in each channel of the 3-channel 8-bit color image data. May be assigned. For example, since each axial component of the normal vector takes a value of -1.0 to 1.0, by assigning a value between them from 0 to 255, the information of the normal vector can be used as 3-channel 8-bit color image data. Can be owned.

In step S902, the normal correction unit 304 converts the face normal image data 1001 using the affine transformation coefficient calculated in step S901 to generate the normal image data 1003. As a result, the normal image data 1003 is generated by fitting the face normal image data 1001 to the face region included in the color image data 501. In the normal image data 1003, the normal vector (N'x (i, j), N'y (i, j), N'z (i, j)) is used as a pixel value for the pixel N'(i, j). ) Is stored in the image data. The normal vector of the normal image data 1003 is a normal vector (Nx, Ny, It is calculated based on Nz). Then, for the region that does not correspond to the face normal image 1001, it is assumed that the normal vector (0, 0, 1) in the direction opposite to the optical axis of the imaging device is stored. By this step, the face area in the face normal image 1001 can be roughly matched with the face area in the color image data. However, positions other than organs such as the contour of the face may not be accurately aligned, so this will be corrected in the subsequent steps.

In step S903, the normal correction unit 304 divides the normal image data 1003 into components in the x-axis, y-axis, and z-axis directions, and divides the normal image data 1003 into x-axis component normal data 1101, y-axis component normal data 1102, and z-axis. It is decomposed into three image data of the component normal data 1103 (FIG. 11 (a)). This makes it possible to apply the same smoothing process as the binary image 803. In the present embodiment, the normal correction unit 304 operates the joint bilateral filter in the same manner as in step S703.

In step S904, the normal correction unit 304 performs a smoothing process on the x-axis component normal data 1101 to generate the smoothed x-axis component normal data 1104. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed x-axis component normal data 1104 obtained by this process stores the smoothed x-axis component value N "x in each pixel.

In step S905, the normal correction unit 304 performs a smoothing process on the y-axis component normal data 1102 to generate the smoothed y-axis component normal data 1105. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed y-axis component normal data 1105 obtained by this processing stores the smoothed y-axis component value N "y in each pixel.

In step S906, the normal correction unit 304 performs a smoothing process on the z-axis component normal data 1103 and generates the smoothed z-axis component normal data 1106. As the smoothing process, a joint bilateral filter using the color image data 501 as a reference image is applied. It is assumed that the smoothed z-axis component normal data 1106 obtained by this process stores the smoothed z-axis component value N "z in each pixel.
From step S904 to step S906, the contour of the face in the normal image data 1003 can be matched with the contour of the subject in the color image data.

In step S907, the normal correction unit 304 integrates the smoothed x-axis component normal data 1104, the smoothed y-axis component normal data 1105, and the smoothed z-axis component normal data 1106, and the smoothed normal image data 1107. Is generated (FIG. 11 (b)). The smoothed normal image data 1107 is an image in which normal vectors (N "x (i, j), N" y (i, j), N "z (i, j)) are stored in pixels (i, j). It is data.

In step S908, the normal vector stored in each pixel of the smoothed normal image data 1107 is normalized so as to be a unit vector. In steps S904 to S906, since the smoothing process is performed for each axis component, the magnitude of the normal vector stored in each pixel is different. In order to correct this, in this step, normalization is performed so that the magnitude of the normal vector becomes 1 as in Eq. (2).

As a result, the normal vector of size 1 (N ′ ″ x (i, j), N ″ ″ y (i, j), N ″ ′ z (i, j) for the pixel (i, j). ) Is stored in the correction normal data.

As described above, the normal correction unit 304 acquires the correction normal data. According to the above processing, since the face normal model can be corrected according to the face of the subject, it is possible to give a natural shadow to the face of the subject in the lighting process. Further, by performing the smoothing process independently for each coordinate axis component as described above, it is possible to prevent the normal direction from being significantly changed by the smoothing process.

<Lighting process>
Here, the lighting process performed in step S411 will be described. The lighting process in the present embodiment is a process of generating a corrected image by adding a virtual light source to the color image data according to the lighting parameters set by the user operation based on the correction distance data and the correction normal data. Is. Hereinafter, the details of the lighting process will be described with reference to the flowchart shown in FIG.

In step S1201, the lighting unit 305 acquires the lighting parameters used for the lighting process set by the user from the control unit 206. In the present embodiment, the user sets the position Q, the posture U, the intensity α, and the light source color L of the virtual lighting as lighting parameters by operating the operation unit 107.

In step S1202, the lighting unit 305 corrects the pixel value of the color image data 501 based on the correction distance data 804, the normal image data 1003, and the illumination parameters acquired in step S1101. In the present embodiment, the pixel value of the color image data is corrected according to the equation (3), and the corrected image data is generated as I'.

Here, I'r, I'g, and I'b are the pixel values of the corrected image data I', Lrm, Lgm, and Lbm are the m-th illumination colors, and km is the correction degree of the pixel values for the m-th illumination. show. km is determined based on the brightness α of the illumination, the position Q, the attitude U, the distance value corresponding to the pixel (x, y), and the normal vector V. For example, it can be obtained by the equation (4).

Equation (4) will be described with reference to FIG. t is a correction coefficient for adjusting the degree of correction by the virtual light source. In this embodiment, t = 1. α is a variable that represents the brightness of the illumination. Q is a vector representing the position of the light source. P is a vector representing the three-dimensional position of the pixel (i, j), and is calculated from the correction distance data 804 as follows. First, based on the pixel value of the correction distance data 804, a virtual distance value from the image pickup apparatus 101 to the subject position corresponding to each pixel is calculated. At this time, it is assumed that the larger the pixel value in the correction distance data 804, the smaller the distance from the image pickup apparatus 101. Subsequently, the lighting unit 305 uses the virtual distance value corresponding to each pixel, the angle of view of the imaging device 101, the image size of the color image data 501, and the like, and the three-dimensional positions P of the pixels (i, j). Is calculated. W is a function that returns a larger value as the distance from the position P of the pixel (i, j) to the position Q of the light source increases. ρ represents the angle formed by the vector from Q to P (i, j) and the attitude U of the illumination. K is a function such that the smaller ρ is, the larger the value is. N (i, j) is a normal vector corresponding to the pixel (i, j), and V (i, j) is a unit vector representing the direction from Q to P (i, j). By generating the corrected image as in the present embodiment, it is possible to correct the brightness according to the position of the illumination and the shape of the subject. As described above, the lighting process of adding the pixel values according to the distance from the virtual light source is performed. By the above processing, it is possible to correct the pixel so that it is closer to the virtual light source and the angle between the vector from the virtual light source toward the pixel (i, j) and the normal vector is smaller, the brighter the pixel. As a result, as shown in FIG. 14, it is possible to obtain a corrected image 1401 as if the subject was illuminated by virtual illumination.

In step S1203, the lighting unit 305 displays the corrected image data in which the pixel value has been corrected on the display unit 106, and ends the process. The user sees the corrected image data displayed on the display unit 106, and inputs an instruction to change the lighting parameter and an instruction to output the image.

According to the above processing, since the lighting processing can be performed using the face normal model deformed according to the subject, it is possible to obtain an image with natural shading according to the posture and facial expression of the subject.

[Embodiment 2]
In the first embodiment, an example of correcting the pixel value of the color image data regardless of the pixel value of the subject is generated. In the second embodiment, a method of controlling the correction amount based on the brightness value of the subject will be described. By controlling the correction amount based on the brightness value of the subject, it is possible to suppress overexposure that occurs when a region having a high brightness value is corrected in advance and noise increase that occurs when a dark part is corrected.

Since the configuration of the image pickup apparatus 101 of the present embodiment and the basic processing flow are the same as those of the first embodiment, the description thereof will be omitted. The difference between the second embodiment and the first embodiment is that the correction coefficient t shown in the equation 4 is determined by the luminance value of each pixel in the lighting process performed by the lighting processing unit 305.

An example of a method for determining the correction coefficient t in the present embodiment will be described with reference to FIG. FIG. 15A shows an example in which the correction coefficient t is determined based on the preset threshold values th1 and th2. In this example, the pixel brightness value Y is corrected so that t = 1 in the section where 0 ≦ Y <th1, t decreases monotonically in the section where th1 ≦ Y <th2, and t = 0 in the section where th2 ≦ Y. The coefficient t is determined. When the correction coefficient t is determined in this way, the degree of correction by the virtual light source can be reduced as the pixel has a larger luminance value. Therefore, it is possible to obtain the effect of suppressing overexposure of pixels having a high luminance value due to the influence of the virtual light source. In FIG. 15A, t is linearly reduced so that t becomes a linear function of Y in the interval of th1 ≦ Y <th2, but the method of reduction is not limited to this. When t is a linear function of Y, the corrected image I'is expressed as a quadratic function of the pixel value of the color image data I. In this case, the pixel value of the corrected image becomes maximum in the interval of th1 ≦ Y <th2, and gradation inversion may occur. As a means of suppressing this, the decrease in the interval of th1 ≦ Y <th2 may be expressed by using a quadratic curve, a trigonometric function, or the like. By doing so, it is possible to suppress the occurrence of gradation inversion in the corrected image. Alternatively, the correction coefficient t can be determined as shown in FIG. 15 (b). FIG. 15B shows an example of the correction coefficient determined by the preset threshold values th1, th2, and th3. In this example, t increases monotonically in the section where the brightness value Y is 0 ≦ Y <th3, t = 1 in the section where th3 ≦ Y <th1, t decreases monotonically in the section where th1 ≦ Y <th2, and th2 ≦ Y. In the interval, the correction coefficient t is determined so that t = 0. By setting the correction coefficient t as shown in FIG. 15B, it is possible to suppress that the noise in the dark portion having a small brightness value in addition to the overexposure is emphasized by the lighting process. Further, by expressing the increase in the interval where the luminance value Y is 0 ≦ Y <th3 and the decrease in the section where th1 ≦ Y <th2 by using a quadratic curve, a trigonometric function, or the like, FIG. As in the case of, the occurrence of gradation inversion can be suppressed.

As described above, according to the processing of the present embodiment, it is possible to suppress overexposure that occurs when a region having a high luminance value is corrected in advance and noise increase that occurs when a dark portion is corrected.

[Embodiment 3]
In the first and second embodiments, an example of performing a lighting process for brightening a dark subject by applying a virtual light source to the scene has been described. In the third embodiment, a method of emphasizing the three-dimensional effect of the subject by adding a shadow to the subject which is projected flat due to the influence of the light emission of the strobe or the like will be described.

Since the configuration of the image pickup apparatus 101 of the present embodiment and the basic processing flow are the same as those of the first embodiment, the description thereof will be omitted. The difference between the third embodiment and the first embodiment is that the pixel value correction process performed in step S1202 is different. Hereinafter, the process performed in step S1202 of the present embodiment will be described. In the present embodiment Nostep S1202, unlike the first embodiment, the pixel value is corrected based on the following equation (5).

The difference from the equation (3) is that the pixel value is corrected so that the pixel value of the color image data becomes smaller according to k'm. That is, what is performed in this embodiment is a lighting process in which the pixel value is subtracted according to the distance from the virtual light source. k'm is determined based on the brightness α of the illumination, the position Q, the attitude U, the distance value corresponding to the pixel (x, y), and the normal vector V. For example, it can be obtained as in Eq. (6).

The difference from the equation (4) is mainly the influence of the angle formed by the normal vectors N (i, j) and V (i, j). In equation (4), the value of k increased as the normal vector N pointed in the direction of the virtual light source, whereas in equation (6), the value of k increased as the normal vector N pointed in the direction of the virtual light source. Becomes smaller. That is, according to the equation (6), a pixel closer to the virtual illumination and the normal vector N does not face the direction of the virtual light source can give a stronger shadow. As a result, as in the corrected image 1601 shown in FIG. 16, it is possible to add shadows only to the cheeks and nose of the face based on the normal image data.

According to the above processing, it is possible to perform correction for adding a shadow to a subject that appears flat due to the influence of light emission of a strobe or the like so as to give a three-dimensional effect.

[Embodiment 4]
In the above embodiment, the rewriting process of giving a virtual light source to the scene and the rewriting process of giving a shadow to the image have been described. In the fourth embodiment, a method of switching between the above two processes based on the shooting conditions will be described. FIG. 17 is a flowchart showing an operation procedure of the image processing unit 209 according to the fourth embodiment. The difference is that step S1701 and step S1702 are newly added as compared with the first embodiment.

In step S1701, the lighting unit 305 acquires information about the actual light source. Here, the actual light source is a light source that actually exists in the space where the subject is imaged. In the present embodiment, the control unit 206 determines whether or not the strobe is emitted based on the user's instruction to use the strobe and the input signal from the strobe 104, and informs the lighting unit 305 whether or not the strobe is used when capturing the image data. Suppose you want to output. If it is determined that the strobe has been used at the time of capturing the image data, it is acquired as if the strobe arranged at the predetermined position Q'is emitting light. The method of acquiring the information of the actual light source is not limited to this. For example, the average brightness of the pixels in the face region of the selected subject is obtained, and if the average brightness is equal to or more than the threshold value, it is assumed that the strobe is emitted at the time of imaging. You may judge.

In step S1702, the lighting unit 305 sets the lighting processing mode based on the actual light source information acquired in step S1701. In this step, if it is determined in step S1701 that the strobe is not emitting light at the time of imaging, a lighting mode for applying the virtual light source shown in the first embodiment is set. Then, in step S1701, when it is determined that the strobe is emitting light at the time of imaging, the lighting mode for imparting the shadow shown in the third embodiment is set.

Then, in step S411, the lighting unit 305 performs the lighting process corresponding to the lighting mode set in step S1702 on the color image data to generate the corrected image data.

The above is the flow of processing in this embodiment. According to the above processing, an appropriate lighting processing can be selected according to the state of the light source when the subject is imaged.

The processing of this embodiment is not limited to the above. For example, the position Q'of the strobe light may be acquired as the actual light source information in step S1701, and the position Q'of the strobe light may be input as the initial value of the lighting parameter in the lighting process of step S411. Further, it is assumed that an actual light source other than the strobe exists in a region where the brightness is larger than a predetermined threshold value in the color image data, and when the detected actual light source exists at a position closer to the image pickup apparatus 101 than the subject. The lighting mode may be set to the shadow addition mode. Alternatively, the position of the actual light source may be acquired from the color image data and input to the initial value of the illumination parameter.

<Other Embodiments>
The embodiment of the present invention is not limited to the embodiment shown above. For example, the lighting process may be performed by directly using the distance information of the subject without using the normal image data in the lighting process. In this case, since it is necessary to use a calculation formula different from the above formula, the processing becomes complicated, but the same effect as that of the present invention can be obtained. At that time, a 3D model of a predetermined face may be held instead of the face normal model. That is, in carrying out the present invention, it is possible to widely use information indicating the three-dimensional shape of the subject. Further, a 3D model of a predetermined face may be held instead of the face normal model, and normal information may be acquired based on the deformed 3D model.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

301 Development unit 302 Distance correction unit 303 Face detection unit 304 Normal correction unit 305 Lighting unit

In order to solve the above problems, the image processing apparatus according to the present invention specifies an acquisition means for acquiring an captured image obtained by imaging a subject and a region of the face of the subject as a face region in the captured image. The specific means for generating the corrected image in which the brightness of at least a part of the face region in the captured image is changed is provided, and the generating means is a region of the face region that does not face the virtual light source. The brightness is changed so that a stronger shadow is given to the pixel corresponding to the above, and the degree of the change is suppressed in the pixel having a relatively small brightness value in the face region.

Claims (10)

An acquisition means for acquiring an image obtained by photographing a subject, and
A specific means for identifying the face area of the subject as a face area in the color image, and
An image processing apparatus comprising a generation means for generating a corrected image in which the brightness of at least a part of the face region in the color image is changed. The image processing apparatus according to claim 1, wherein the generation means generates an image whose brightness is changed according to the shape of the face as the correction image. In the generation means, the degree to which the brightness in the face region is changed is the degree to which the brightness in the subject for which the specific means has specified the face region and the background region excluding the region in the vicinity of the subject is changed. The image processing apparatus according to claim 1, wherein the corrected image is generated so as to be larger than the above. The image processing apparatus according to claim 1, wherein the specifying means specifies only the face of a person as the face region among the subjects in the color image. The image processing apparatus according to claim 1, wherein the generation means imparts a shadow according to a shape in the face region. Claim 1 is characterized in that the degree of change from the brightness of the face region in the color image to the brightness of the face region in the corrected image depends on the brightness of the face region in the color image. The image processing apparatus according to. In the corrected image, the distance between the first pixel included in the face region and the image pickup device included in the face region and the image pickup device in the first pixel is the same, and the distance from the image pickup device is the same, and the first pixel A claim characterized in that, for a second pixel corresponding to a normal direction different from the normal direction, the degree of change from the brightness of each of the first pixel and the second pixel in the color image is different. Item 6. The image processing apparatus according to any one of Items 1 to 6. The first aspect of the present invention is characterized in that the generation means does not change the brightness according to the normal direction of the subject in a region other than the face region specified by the specific means and the vicinity of the face region. The image processing apparatus described. A program that causes a computer to function as the image processing device according to any one of claims 1 to 8. It is an image processing method that generates a corrected image in which the brightness of a color image obtained by photographing a subject is changed.
Acquire the color image and
In the color image, the face area of the subject is specified as the face area, and the face area is specified.
An image processing method for generating the corrected image in which the brightness of at least a part of the face region in the color image is changed.

Images