JP7057086B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

Face orientation, composition, lighting, etc. are one of the major factors that determine the impression of a subject in photography of a person. Therefore, the subject is corrected by adjusting these shooting conditions after taking a picture. For example, Patent Document 1 discloses an image processing device that corrects a subject by using a face model generated from a face three-dimensional shape template representing a three-dimensional shape of a face.

In the image processing apparatus of Patent Document 1, the face three-dimensional shape template is deformed based on the face information (that is, face position information, face size information, face component information, and face direction information) detected from the image data. To generate a face model of the subject. Further, the image processing apparatus of Patent Document 1 estimates the three-dimensional shape of the face based on the detected face component information, face size information, and the like, and from a plurality of face three-dimensional shape templates, a face three-dimensional shape template suitable for the subject. Is selected. This makes it possible to select a three-dimensional face shape template according to individual differences in the face and differences in appearance due to the orientation of the face and the like.

Japanese Unexamined Patent Publication No. 2014-49866

However, when a face three-dimensional shape template is selected without considering at least the subject distance as in the image processing device of Patent Document 1, when the subject distance changes, a face three-dimensional shape template different from the appearance of the subject is selected. There is a possibility. As a result, the face model is also generated so as not to resemble the subject, and as a result, the subject corrected by using the face model is generated as unnatural.

The present invention has been made in view of the above-mentioned conventional problems, and an object thereof is to generate an image as if it was taken under desired lighting conditions from a captured image even if the subject distance changes. Is.

In order to achieve the above object, the image processing device of the present invention has an image acquisition means for acquiring image data obtained by capturing a scene including a subject, and a distance from the image pickup device that has captured the scene to the subject. Based on the first acquisition means for acquiring information, the storage means for storing normal information corresponding to a predetermined feature pattern related to the subject, and the distance information and the normal information stored in the storage means. A generation means for generating normal information applied to the lighting process for the image data, and a lighting processing means for executing the lighting process for the image data using the normal information generated by the generation means. The storage means stores a plurality of normal information corresponding to a predetermined feature pattern according to the distance from the image pickup device to the subject, and the generation means stores a plurality of methods stored by the storage means. Using the normal information selected from the line information based on the distance information, the normal information to be applied to the lighting process is generated .

According to the present invention, even if the subject distance changes, it is possible to generate an image as if it was taken under desired lighting conditions from the captured image.

It is a figure which shows the appearance of the image pickup apparatus provided with an image processing apparatus. It is a block diagram which shows the internal structure of the image pickup apparatus provided with an image processing apparatus. It is a block diagram which shows the functional structure of an image processing apparatus. It is a flowchart which shows the processing procedure in an image processing apparatus. It is a figure which shows the color image data and the distance image data. It is a figure which shows the face area and the organ position. It is a flowchart which shows the procedure of the face model normal information acquisition processing. It is a figure which shows the face model normal information. It is a flowchart which shows the procedure of the normal image data generation processing. It is a figure which shows the outline of the normal image data generation processing. It is a figure which shows the outline of a lighting process. It is a figure which shows the result of a lighting process. It is a flowchart which shows the procedure of the face model normal information acquisition processing. It is a figure which shows the face model normal information by area. It is a figure which shows the face model normal information. It is a flowchart which shows the procedure of the face model normal information acquisition processing by area. It is a figure which shows the outline of the face normal image data generation processing. It is a block diagram which shows the functional structure of an image processing apparatus. It is a flowchart which shows the processing procedure in an image processing apparatus. It is a figure which shows the subject model normal information.

(Appearance of image pickup device)
1A and 1B are views showing the appearance of an image pickup apparatus including the image processing apparatus according to the embodiment, FIG. 1A shows the appearance of the front surface of the image pickup apparatus, and FIG. 1B shows the appearance of the back surface. .. The image pickup apparatus 101 includes an optical unit 102, an image pickup button 103, a strobe 104, a distance image acquisition unit 105, a display unit 106, and an operation button 107.

The optical unit 102 is a lens barrel composed of a zoom lens, a focus lens, a blur correction lens, an aperture, and a shutter, and collects light information of a subject. The image pickup button 103 is a button for the user to instruct the image pickup apparatus 101 to start imaging. The strobe 104 is an illumination capable of emitting light at the start of imaging according to a user's instruction.

The distance image acquisition unit 105 acquires the distance image data of the subject in the scene captured in response to the imaging instruction as the distance information. Here, the distance image data is image data in which the subject distance corresponding to the pixel is stored as the pixel value of each pixel of the image. The distance image acquisition unit 105 includes an infrared light emitting unit that emits infrared light and a light receiving unit that receives infrared light reflected by the subject. The distance image acquisition unit 105 calculates the distance value from the image pickup apparatus to the subject based on the time until the infrared light emitted by the infrared light emitting unit is reflected by the subject and the light receiving unit receives the light. Then, the position information of the subject is calculated based on the calculated distance value and the distance imaging information including the number of sensor pixels of the light receiving unit, the angle of view, and the like, and the distance image data is generated. The method of acquiring the distance image data is not necessarily limited to this. Therefore, for example, instead of the distance image acquisition unit 105, an optical system similar to the optical unit 102 is provided, and the distance image data is obtained by performing triangulation based on the parallax between the image data captured from two different viewpoints. You can also get it.

The display unit 106 is a display device such as a liquid crystal display that displays image data processed by the image pickup device 101 and various other data. As shown in FIG. 1, since the image pickup apparatus 101 does not have an optical finder, the framing operation (confirmation of focus and composition) is performed using the display unit 106. That is, since the image pickup device 101 is imaged while confirming the live view image on the display unit 106, the display unit 106 also functions as an electronic finder when the framing or focusing operation is performed. In addition, the display unit 106 also displays a camera setting menu and the like.

The operation button 107 is a button for the user to instruct the image pickup device 101 of an operation for switching the operation mode of the image pickup device 101, various parameters at the time of image pickup, and the like. The image pickup apparatus 101 includes, as one of the operation modes, a lighting processing mode for correcting the lighting condition of the captured image after imaging. Therefore, the user uses the operation button 107 or the image pickup button 103 to switch to the lighting processing mode, set the lighting parameters of the virtual lighting used for the lighting processing, and select the subject for which the lighting condition is adjusted (corrected). It can be performed. Further, when outputting the corrected image data, the user can also instruct whether or not to output the distance image data. In addition, the display unit 106 may have a touch screen function, and in that case, a user instruction using the touch screen can be handled as an input of the operation button 107.

(Internal configuration of image pickup device)
FIG. 2 is a block diagram showing an internal configuration of an image pickup apparatus including the image processing apparatus according to the embodiment. The CPU 201 is involved in all the processes of each configuration, and executes the processes by sequentially reading and interpreting the instructions stored in the ROM (Read Only Memory) 202 and the RAM (Random Access Memory) 203.

The ROM 202 stores a program executed by the CPU 201. In this embodiment, the ROM 202 stores a plurality of face model normal information and area-specific face model normal information. The face model normal information includes face model normal image data in which the normal vector of the face surface corresponding to a face (face model) having a predetermined shape is stored in pixel values, and eyes / nose in the face model normal image data. -Includes face model normal organ positions that indicate organ positions such as the mouth. The area-specific face model normal information includes area-specific face model normal image data in which normal vectors corresponding to areas such as face contour, eyes, nose, and mouth are stored in pixel values, and area-specific face model normals. Includes the area-specific face model normal organ position indicating the organ position in the image data. The RAM 203 stores a program executed by the CPU 201.

The optical system control unit 204 is a control circuit that controls the optical unit 102 instructed by the CPU 201, such as focusing, opening a shutter, and adjusting an aperture. Further, the optical system control unit 204 can acquire the distance to the focused subject and store it in the RAM 203 or the like based on the state of the optical system or the like.

The control unit 205 is a control circuit that receives user instructions from the image pickup button 103 and the operation button 107, and controls switching to an image pickup mode, a lighting processing mode, and the like, selection of a subject area, setting of lighting parameters, and the like. The color image sensor unit 206 is an image sensor that converts the light information collected by the optical unit 102 into a current value. The color image sensor unit 206 includes a color filter having a predetermined arrangement such as a bayer arrangement, and acquires color information of a subject from the light focused by the optical unit 102.

The A / D conversion unit 207 is a processing circuit that converts the color information of the subject detected by the color image pickup element unit 206 into a digital signal value and converts it into RAW image data. In this embodiment, it is assumed that the distance image data and the RAW image data captured at the same time can be acquired. The image processing unit 208 performs development processing on the RAW image data acquired by the A / D conversion unit 207 to generate color image data. Further, the image processing unit 208 executes various image processing such as generating corrected image data obtained by applying lighting processing to the color image data using the color image data and the distance image data. The internal configuration and function of the image processing unit 208 will be described in detail with reference to FIG. 3 described later.

The character generation unit 209 is a processing circuit that generates characters, graphics, and the like. The characters and graphics generated by the character generation unit 209 are superimposed and displayed on the image data, the corrected image data, and the like on the display unit 106. The encoder unit 210 converts various image data including the color image data processed by the image processing unit 208 and the corrected image data generated by the lighting processing into a file format such as JPEG.

The media I / F211 is an interface for transmitting / receiving image data to / from a PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.). As the media I / F211th, for example, USB (Universal Serial Bus) or the like is used. The system bus 212 is a bus for transmitting and receiving data.

(Internal configuration of image processing unit)
FIG. 3 is a block diagram showing a functional configuration of the image processing apparatus (image processing unit 208) according to the embodiment. The development processing unit 301 performs white balance processing, demosaic processing, noise reduction processing, color conversion processing, and the like on the RAW image data acquired from the A / D conversion unit 207 to generate color image data. The image pickup apparatus 101 can output the color image data generated by the development processing unit 301 to the display unit 106 for display, or can store the color image data in a storage device such as a RAM 203 or a PC / media 213.

The face detection unit 302 acquires the face information of the subject from the color image data acquired from the development processing unit 301. The face information of the subject includes at least information on a face area indicating an area occupied by the face of the subject in the color image data and an organ position indicating a position in the color image data such as eyes and mouth included in the face. ..

The face model normal information acquisition unit 303 selects normal information suitable for the subject from a plurality of face model normal information based on the distance information to the subject. In the present embodiment, the face model normal information is selected from the ROM 202 based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105.

When the normal line generation unit 304 acquires face information from the face detection unit 302, face model normal line information from the face model normal line information acquisition unit 303, and distance image data from the distance image acquisition unit 105, it is based on the information. Then, the normal image data corresponding to the color image data is generated.

The lighting parameter setting unit 305 sets the lighting parameter based on the user operation (instruction) passed from the control unit 205. The lighting parameters in the present embodiment include the position, posture, intensity, light source color, and the like of the lighting. The lighting processing unit 306 acquires distance image data from the distance image acquisition unit 105, normal image data from the normal line generation unit 304, and lighting parameters from the lighting parameter setting unit 305, and a color image based on these data. Perform a writing process on the data. The image pickup device 101 can output and store the corrected image data corrected by the lighting process to a storage device such as a RAM 203 or a PC / media 213, or can output the corrected image data to a display unit 106 for display.

(Processing flow of image processing unit)
FIG. 4 is a flowchart showing a processing procedure in the image processing apparatus (image processing unit 208) according to the embodiment. In the process shown in the flowchart of FIG. 4, the image processing unit 208 first selects the face model normal information based on the face information and the distance image data acquired from the color image data. Next, the image processing unit 208 generates normal image data corresponding to the color image data based on the face information, the face model normal information, and the distance image data. The image processing unit 208 further executes a lighting process for adding virtual lighting to the color image data based on the lighting parameters, the distance image data, and the normal image data set by the user, and generates the corrected image data.

Hereinafter, the processing procedure of the image processing unit 208 will be described in detail. The development processing unit 301 generates color image data based on the RAW image data acquired from the A / D conversion unit 207 (S401). The color image data is shown, for example, as shown in FIG. 5A, and RGB values are stored as pixel values in the pixels I (i, j) of the color image data 501, respectively, and Ir (i, i, j), Ig (i, j), Ib (i, j). The method for acquiring color image data is not necessarily limited to this. Therefore, for example, RAW image data stored in the RAM 203 or the PC / media 213 can be acquired, and the development processing unit 301 can generate color image data from the acquired RAW image data. It is also possible to directly acquire the color image data stored in the RAM 203 or the PC / media 213.

The image processing unit 208 acquires distance image data from the distance image acquisition unit 105 (S402). The distance image data is shown, for example, as shown in FIG. 5B, and the distance value from the image pickup apparatus to the subject is stored in the pixels D (i, j) of the distance image data 502. The method of acquiring the distance image data is not necessarily limited to this. Therefore, for example, the distance image data stored in the RAM 203 or the PC / media 213 can be directly acquired.

Next, the image processing unit 208 acquires the state of the flag SW1 from the RAM 203, and determines whether or not to execute the writing process based on the state of the flag SW1 (S403). The image processing unit 208 determines that the lighting process is not executed when the flag SW1 is OFF (S403 No), and shifts the process to step S404. Further, when the flag SW1 is ON, it is determined that the lining process is executed (S403 Yes), and the process shifts to step S405.

If it is determined in step S403 that the lighting process is not executed, the image processing unit 208 acquires the state of the flag SW2 from the RAM 203. Then, the image processing unit 208 outputs and stores only the color image data when the flag SW2 is OFF, and the color image data and the distance image data when the flag SW2 is ON to the PC / media 213 (S404). Alternatively, the image processing unit 208 can output the color image data to the display unit 106 and display it so that the user can confirm it. When the image processing unit 208 executes the process in step S404, the image processing unit 208 ends the process shown in FIG.

Further, when it is determined in step S403 that the lighting process is to be executed, the face detection unit 302 detects (acquires) face information based on the color image data (S405). Here, the face information will be described with reference to FIG. As shown in FIG. 6, the face information includes the face area 601 and the organ position 602. The face region is shown in the image data as a set of pixels in the region including the face. The organ position 602 is shown as coordinates corresponding to the eyes, nose, mouth, etc. in the facial region.

Existing algorithms can be applied to the method of detecting the facial region and organ position. As the existing algorithm, there are an algorithm using template matching, an algorithm using Haar-Like feature amount, and the like. In this embodiment, the face region / organ position is detected by applying template matching.

In the detection to which the template matching is applied, first, the skin color region is extracted as the face candidate region by executing the threshold value processing on the color image data. Next, using face image templates of various sizes, matching processing is executed for the face candidate area, and the likelihood is calculated. Then, based on the calculated likelihood, a process of determining whether or not the face area is present is executed, and the face area is extracted. With regard to the organ positions, the matching process is similarly executed for the extracted face area using the eye, nose, and mouth image templates, and the likelihood is calculated. Then, the reliability of the facial region / organ position is calculated based on the magnitude of this likelihood. By executing the above processing, the face area 601 and the organ position 602 are acquired.

Next, the face model normal information acquisition unit 303 acquires face model normal information based on the subject distance (S406). The process of acquiring the face model normal information in the face model normal information acquisition unit 303 will be described in detail with reference to FIG. 7 and the like described later.

The normal generation unit 304 corresponds to the color image data generated in step S401 based on the distance image data, face information, and face model normal information acquired in steps S402, S405, and S406, respectively. Generate line image data (S407). The process of generating normal image data in the normal generation unit 304 will be described in detail with reference to FIGS. 9, 10 and the like described later.

The lighting parameter setting unit 305 sets (determines) the lighting parameter based on the user operation passed from the control unit 205 (S408). The lighting processing unit 306 executes lighting processing such as adding virtual lighting (virtual light source) to the color image data for the color image data based on the distance image data, the normal image data, and the lighting parameters. Then, the corrected image data is generated (S409). The lighting process in the lighting process unit 306 will be described in detail with reference to FIG. 11 and the like described later.

The writing processing unit 306 determines whether or not to end the writing processing based on the user operation (that is, an instruction from the user) passed from the control unit 205 (S410). The image processing unit 208 shifts the processing to step S411 when it is determined by the lighting processing unit 306 to end the lighting processing, returns the processing to step S408 when it is determined not to end the lighting processing, and returns the lighting parameter again. To set.

When the processing shifts to step S411, the image processing unit 208 acquires the state of the flag SW2 from the RAM 203, and the lighting processing unit 306 outputs various data based on the acquired state of the flag SW2. The lighting processing unit 306 outputs color image data and corrected image data when the flag SW2 is OFF, and outputs color image data, corrected image data, and distance image data when the flag SW2 is ON to the PC / media 213. Remember. After that, the image processing unit 208 ends the process shown in FIG.

(Face model normal information acquisition process)
Here, regarding the face model normal information acquisition process (that is, the process of acquiring the face model normal information based on the distance image data and the face information) executed by the face model normal information acquisition unit 303 in step S406. explain.

Hereinafter, the procedure of the face model normal information acquisition process will be described in detail with reference to FIG. 7. The face model normal information acquisition unit 303 first calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105 (S701). In the present embodiment, the average of the distance values in the face area (that is, the area occupied by the face of the subject) is taken as the subject distance.

The face model normal information acquisition unit 303 acquires face model normal information based on the subject distance calculated in step S701 (S702). In the present embodiment, the face model normal information closest to the subject distance calculated in step S701 is selected from the plurality of face model normal information corresponding to the subject distance shown in FIG.

FIG. 8 illustrates a plurality of face model normal information according to the subject distance. In FIG. 8, the face model normal information 801, 802, and 803 are face model normal information corresponding to the subject distances of 0.5 m, 1.5 m, and 5.0 m, respectively, and these face model normal information are It is stored in ROM 202.

The face model normal information includes face model normal image data in which the normal vector of the face surface (that is, the orientation of the face surface when observed from the position of the subject distance) is stored in pixel values, and face model normal image data. Includes the face model normal organ position indicating the position of organs such as eyes, nose, and mouth in. It should be noted that the pixel value indicating the background pixel is stored in the pixel (that is, the background pixel) in the region other than the face region in the face model normal image data.

As shown in FIG. 8, in the case of a frontal face, the shorter the subject distance, the farther the organ positions such as eyes and mouth are from the center (that is, closer to the facial contour), and the longer the subject distance, the more the organ positions such as eyes and mouth. Looks like it's closer to the center of the face. In this way, the balance between the facial contour and the eyes, nose, mouth, etc. changes depending on the subject distance, and the change in the organ position increases as the subject distance becomes shorter. Therefore, regarding the face model normal information, it is preferable to retain the face model normal information at shorter distance intervals as the subject distance is shorter.

Further, in FIG. 8, the case where the face model normal information is three is described as an example, but the number of face model normal information is not limited to this. Therefore, for example, the number of face model normal information may be set to two or four or more depending on the capacity of the ROM 202 or the like. If a large number of face model normal information is set, it is possible to select face model normal information that is more suitable for the conditions at the time of shooting.

(Normal image data generation processing)
Here, in step S407, the normal image data generation process executed by the normal generation unit 304 (that is, the normal image corresponding to the color image data based on the face information, the face model normal information, and the distance image data). The process of generating data) will be described. Hereinafter, the procedure and outline of the normal image data generation process will be described in detail with reference to FIGS. 9 and 10, respectively.

The normal generation unit 304 generates subject normal image data based on the distance image data (S901). Regarding the generation of the subject normal image data of the present embodiment, the normal generation unit 304 first obtains the two-dimensional coordinate values of the pixels in the distance image data and the distance values obtained from the pixel values of the pixels. , Calculate three-dimensional coordinate values for each pixel. Then, the normal generation unit 304 then calculates a normal vector corresponding to each pixel based on the calculated coordinate values. The normal vector corresponding to the pixel of interest is calculated based on the gradient calculated from the three-dimensional coordinate values of the region near the pixel of interest. In this way, the normal generation unit 304 generates the subject normal image data 1001.

Next, the normal generation unit 304 generates face normal image data based on the face information and the face model normal information (S902). Regarding the generation of the face normal image data of the present embodiment, the normal generation unit 304 uses the face model based on the organ position 602 included in the face information and the face model normal organ position included in the face model normal information. By transforming the normal image data, the face normal image data 1002 is generated. In this embodiment, the affine transformation is used to transform the face model normal image data so as to match the face of the subject in the color image data 501. Specifically, the face method is performed by calculating the affine parameters from the organ position 602 and the face model normal organ position using the least squares method, and transforming the face model normal image parameters based on the calculated affine parameters. Generates line image data 1002. In the face normal image data, the background pixels other than the face store the pixel values indicating the background pixels.

When the normal line generation unit 304 generates the subject normal image data 1001 and the face normal image data 1002, the normal image data is generated by integrating them and further smoothing them (S903). Specifically, first, the pixels (that is, the background pixels) in the area other than the face area of the face normal line image data 1002 are the pixel values of the subject normal line image data 1001, and the other pixels are the pixel values of the face normal line image data 1002. Image data is integrated so that it becomes a pixel value. That is, the integrated normal image data 1003 is generated. Next, the normal generation unit 304 generates the normal image data 1004 by executing a smoothing process on the generated integrated normal image data 1003. In this embodiment, the color image data 501 is used as a reference image, and the smoothing process is applied to the integrated normal image data 1003. Further, in this case, for example, a joint bilateral filter using the color image data 501 as a reference image may be applied.

By processing as described above, normal image data can be generated. The process of generating normal image data is not necessarily limited to the above-mentioned process. Therefore, for example, it is possible to generate the normal image data only from the face normal image data 1002 without generating the subject normal image data 1001. Further, in this case, instead of the subject normal image data 1001, image data in which an appropriate normal vector is stored may be given to all pixels.

(Lighting process)
Here, the lighting process executed by the lighting processing unit 306 in step S409 will be described. As described above, the lighting process in step S409 adds virtual illumination to the color image data based on the distance image data and the normal image data, and further according to the illumination parameter set by the user, to obtain a corrected image. It is a process to generate. In the present embodiment, the corrected image I'generated by the lighting process is generated according to, for example, the following equation (1).

In the formula (1), I'r, I'g, and I'b are the pixel values of the corrected image data I', Lrm, Lgm, and Lbm are the colors of the m-th illumination, and km is the correction degree for the m-th illumination. show. Further, the correction degree km for the m-th illumination is determined based on the position Q of the virtual illumination, the intensity (brightness of the illumination) α, the distance value corresponding to the posture U and the pixels (x, y), and the normal vector N. Then, for example, it is calculated by the following equation (2). The color L, the position Q, the intensity α, and the posture U of the virtual lighting are lighting parameters set by the user. Further, t is a correction coefficient for adjusting the degree of correction by virtual lighting, and is set to 1 in the present embodiment.

Hereinafter, the description of the formula (2) will be supplemented with reference to FIG. In FIG. 11, Q indicates the position of the virtual illumination as described above, and P indicates the three-dimensional position of the pixel (i, j). In addition, P is the distance value (that is, the distance value of each pixel) from the image pickup apparatus 101 to each pixel acquired based on the pixel value of the distance image data 502, the angle of view of the image pickup apparatus 101, and the color image data 501. It is calculated based on the image size of. Further, W is a function that returns a larger value as the distance from the position P of the pixel (i, j) to the position Q of the virtual illumination increases. That is, the function W corrects the pixels (i, j) so that they become brighter as they are closer to the virtual illumination.

In the equation (2), K is a function that returns a larger value as ρ becomes smaller, and ρ is a vector V (i, j) from the position P of the pixel (i, j) toward the position Q of the virtual illumination. , Shown by the angle formed by the lighting posture U. Further, N (i, j) is a normal vector corresponding to the pixel (i, j). In addition, since the inner product of the vector V (i, j) and the normal vector N is included in the molecule of the equation (2), the vector V toward the virtual illumination from the pixel P (i, j) and the normal vector N The smaller the angle between the pixels, the brighter the pixel. By making such corrections, for example, as shown in FIG. 12, the color image data 501 can be corrected into a corrected image 1201 as if the subject was illuminated by virtual lighting. That is, as shown in the above equation, by executing the lighting process (by generating the corrected image), it is possible to correct the brightness according to the position of the illumination and the shape of the subject.

As described above, in the present invention, even if the subject distance changes, another image as if it was taken under desired lighting conditions can be generated from the captured image. Specifically, even if the subject is about the same size on the captured image, the subject is shot at a long subject distance using a lens with a long focal length, and the subject is short using a lens with a short focal length. The lighting process produces different effects depending on whether the image was taken at a distance.

In the present embodiment, the face normal line information has been described as a specification acquired from the ROM 202 based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105. The data acquisition method is not necessarily limited to this. Therefore, for example, the face normal information can be selected in step S702 based on the subject distance acquired when the optical system control unit 204 focuses on the subject in step S701. If the RAW image data stored in the RAM 203 or the PC / media 213 includes the subject distance as tag information, the subject distance is acquired in step S701, and the face normal information is obtained in step S702. You can also choose.

(Second Embodiment)
In the first embodiment, face model normal information suitable for the subject is selected from a plurality of face model normal information according to the subject distance, but in the present embodiment (second embodiment), the face model normal information is selected according to the subject distance. Generate face model normal information from area-specific face model normal information. This makes it possible to acquire face model normal information corresponding to various subject distances.

FIG. 14 shows the face model normal information for each region as an example. The region-specific face model normal information includes the region-specific face model normal image data 1401 to 1405 shown in FIG. 14 (a) and the region-specific face model normal organ positions shown in FIG. 14 (b).

The face model normal image data 1401 to 1405 for each region shown in FIG. 14 (a) have the normal shape of the face surface in the outline shape of the face, the region around the right eye, the region around the left eye, the region around the nose, and the region around the mouth, respectively. It is the image data stored in the pixel value.

Further, as shown in FIG. 14 (b), the face model normal image data 1402 to 1405 for each region corresponding to the right eye peripheral region, the left eye peripheral region, the nose peripheral region, and the mouth peripheral region are the reference of each organ. An organ reference position indicating the position to be used is set. That is, the region-specific face model normal organ position indicates the position of the organ corresponding to the subject distance in the region-specific face model normal image data 1401 corresponding to the approximate shape of the face. In the example shown in FIG. 14, it is assumed that the face model normal organ positions for each region corresponding to the subject distances of 0.5 m, 1.5 m, and 5.0 m are held (that is, the regions for each subject distance). Another face model normal organ position is stored).

In the second embodiment, the acquisition process of the face model normal information in step S406 is different from that in the first embodiment. FIG. 13 is a flowchart showing a procedure for acquiring face model normal information in the image processing apparatus (image processing unit 208) according to the second embodiment of the present invention.

Hereinafter, the procedure of the face model normal information acquisition process will be described in detail with reference to FIG. The face model normal information acquisition unit 303 calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105, as in step S701 of the first embodiment. (S1301).

When the face model normal information acquisition unit 303 calculates the subject distance, the face model normal organ position corresponding to the subject distance acquired in step S1301 is determined based on the region-specific face model normal organ position shown in FIG. Calculated by interpolation processing (S1302).

Here, the explanation is supplemented with respect to the calculation of the facial model organ position. When the acquired subject distance is l, the region corresponding to the subject distance lA that is smaller than the acquired subject distance l and is the closest to l among the subject distances for which the face model normal organ positions for each region are set. Another face model Normal Select the coordinates of the organ position (xA, yA). Similarly, among the subject distances for which the region-specific face model normal organ positions are set, the region-specific face model normal organ positions corresponding to the subject distance lB that is larger than or closest to l, which is the acquired subject distance. Select the coordinates (xB, yB) of. Then, in this case, the face model organ position (x', y') can be obtained by the following equation (3).

In the formula (3), w is a weight calculated from l, lA, and lB, and can be obtained as in the following formula (4), for example.

In the formula (4), γ is a parameter for adjusting the weight w, and is an interval of the subject distance (for example, 0.1 m, 0.5 m, 1.0 m) in which the face model normal organ position for each region is set. Etc.) can be adjusted according to. Regarding γ, when the interval of the subject distance for which the face model normal organ position for each region is set is shorter than the predetermined value, it is set to 1, and when the interval is longer than the predetermined value. Is set to a value greater than 1. By setting in this way, the shorter the subject distance, the larger the movement amount of the organ, and the longer the subject distance, the smaller the movement amount of the organ.

The face model normal information acquisition unit 303 generates face model normal image data based on the area-specific face model normal image data and the calculated face model organ position (S1303). Specifically, for the face model normal image data 1401 for each region corresponding to the approximate shape of the face, the face model normal image data 1402 to other regions so that the face model organ position and the organ reference position match. Align 1405. Then, the face model normal image data is generated by overwriting the pixel values of the pixels having the normal information of the area-specific face model normal image data 1402 to 1405 with respect to the area-specific face model normal image data 1401. do. FIG. 15 shows, as an example, the face model normal image data 1501 generated by the process shown in FIG.

As described above, in the present invention, even if the subject distance changes, another image as if it was taken under desired lighting conditions can be generated from the captured image. In the present embodiment, as the area-specific face model normal information, one area-specific face model normal image data is held for each area regardless of the subject distance, but different subject distances are supported. A plurality of area-specific face model normal image data may be retained. Therefore, for example, the area-specific face model normal image data corresponding to the subject distances of 0.5 m, 1.5 m, and 5.0 m are retained, respectively, and correspond to the subject distance closest to the subject distance calculated in step S1301. It is also possible to acquire face model normal image data for each area. It is also possible to change the number of face model normal image data for each region held for each region (organ). For example, multiple face model normal image data for each region corresponding to the outline shape of the face and the region around the nose are retained, and the face model normal image data for each region corresponding to the region around the right eye, the region around the left eye, and the region around the mouth is one. Only the sheets may be held.

(Third Embodiment)
In the second embodiment, the face model normal information is generated based on the area-specific face model normal information, but in the present embodiment (third embodiment), the area-specific face model normal is generated according to the subject distance. Select information and generate face normal image data.

In the third embodiment, step S406 is different depending on the relationship (comparison) with the first embodiment, and instead of the face model normal information acquisition process in step S406, the area-specific face model normal information acquisition process is executed. .. Further, the processing content of the face normal image data generation processing in step S902 is different from that of the first embodiment. Hereinafter, these processes (that is, the process of acquiring the face model normal information for each region and the process of generating the face normal image data) will be described in detail.

(Face model normal information acquisition process by area)
Here, the area-specific face model normal information acquisition process executed by the face model normal information acquisition unit 303 will be described. In the present embodiment, regarding the acquisition process of the area-specific face model normal information, the area-specific face model normal information is acquired based on the distance image data and the face information. FIG. 16 shows a procedure for acquiring face model normal information for each region.

The face model normal information acquisition unit 303 calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105, as in step S701 of the first embodiment. (S1601).

The face model normal information acquisition unit 303 selects the area-specific face model normal information corresponding to the acquired subject distance (S1602). In the process of step S1602, the face model normal information for each region closest to the acquired subject distance is selected. For example, in the case of the area-specific face model normal information shown in FIG. 14, the area-specific face model normal image data 1401 to 1405 and the area-specific face model normal organ positions that are closest to the acquired subject distance. To get.

(Face normal image data generation processing)
Here, the face normal image data generation process executed by the normal generation unit 304 will be described. In the present embodiment, the face normal image data is based on the region-specific face model normal information (that is, the region-specific face model normal image data 1401 to 1405 and the region-specific face model normal organ position) acquired in step S1602. To generate. FIG. 17 shows an outline of the face normal image data generation process.

First, the normal generation unit 304 transforms the face model normal image data for each region so as to match the face of the subject in the color image data 501 by using the affine transformation. The affine parameters of the region-specific face model normal image data 1401 corresponding to the facial outline shape are calculated using the least squares method based on the organ position 602 and the region-specific face model normal organ position selected in step S1602. To.

Next, the affine parameters of the other area-specific face model normal image data 1402 to 1405 are translated so that the organ position 602 and the organ reference position corresponding to the area-specific face model normal image data 1402 to 1405 match. Calculated assuming that the components have been corrected. As a result, the area-specific face model normal image data 1402 to 1405 can be deformed according to the organ position 602 of the subject of the color image data 501.

Then, the normal generation unit 304 transforms the area-specific face model normal image data 1401 to 1405 using the conversion parameters calculated as described above, and generates the area-specific face normal image data 1701 to 1705. Further, the normal generation unit 304 generates the face normal image data 1706 by integrating the area-specific face normal image data 1701 to 1705.

That is, in the present embodiment, the pixels having the normal information in the area-specific face normal image data 1702 to 1705 other than the face outline shape are overwritten with the area-specific face normal image data 1701 corresponding to the face outline shape. , Face normal image data 1706 is generated. In all of the area-specific face normal image data 1702 to 1705, the background pixels (that is, the pixels having no normal information related to the area) are the area-specific face normal images corresponding to the outline shape of the face. Face normal image data 1706 is generated with the pixel value of the data 1701.

As described above, in the present invention, even if the subject distance changes, another image as if it was taken under desired lighting conditions can be generated from the captured image. In the present embodiment, the face model normal organ position for each region closest to the acquired subject distance was selected in step S1602, but as described above in step S1302 of the second embodiment, the region is processed by interpolation processing. Another face model normal The organ position may be calculated.

Further, in the present embodiment, as the area-specific face model normal information, one area-specific face model normal image data is held for each area, but the area-specific face model normal corresponding to a different subject distance is described. A plurality of image data may be retained. Therefore, for example, the area-specific face model normal image data corresponding to the subject distances of 0.5 m, 1.5 m, and 5.0 m are retained, respectively, and correspond to the subject distance closest to the subject distance calculated in step S1601. It is also possible to acquire face model normal image data for each area.

In addition, the number of face model normal image data for each region held for each region (organ) can be changed. For example, multiple face model normal image data for each region corresponding to the outline shape of the face and the region around the nose are retained, and the face model normal image data for each region corresponding to the region around the right eye, the region around the left eye, and the region around the mouth is one. Only the sheets may be held.

(Fourth Embodiment)
From the first embodiment to the third embodiment, the processing target is mainly described as a person, but in the present embodiment, in addition to the person (more specifically, the face of the person), other subjects are also processed targets. .. This makes it possible to acquire normal information according to the subject distance for various subjects.

(Internal configuration of image processing unit)
FIG. 18 is a block diagram showing a functional configuration of an image processing apparatus (image processing unit 208) according to an embodiment (fourth embodiment) of the present invention. In the functional configuration of the image processing unit 208 shown in FIG. 18, in relation to the image processing device (image processing unit 208) according to the first embodiment, the subject model normal information is used instead of the face model normal information acquisition unit 303. The acquisition unit 1801 is mounted. In addition, the processing content of the normal generation unit 304 is also different.

The subject model normal information acquisition unit 1801 selects normal information suitable for the subject from a plurality of subject model normal information based on the distance information to the subject and the subject information. In the present embodiment, the subject information selected by the user operation is acquired from the control unit 205. The subject information represents the type of subject included in the color image data, and can be selected from, for example, a dog, a cat, or the like in addition to a person. Further, in this case, it is possible to target only a subject other than a person, such as a dog or a cat.

(Processing flow of image processing unit)
FIG. 19 is a flowchart showing a processing procedure in the image processing apparatus (image processing unit 208) according to the embodiment of the present invention. The flowchart shown in FIG. 19 differs from the processing of steps S1905 to S1907 in relation to (comparison) with the flowchart shown in FIG.

Hereinafter, each process of steps S1905 to S1907 will be described. When it is determined in step S1903 of FIG. 19 that the lighting process is to be executed, the subject model normal information acquisition unit 1801 acquires subject information based on the user operation from the control unit 205 (S1905). The subject model normal information acquisition unit 1801 acquires the subject model normal information corresponding to the subject of the color image data 501 from a plurality of subject model normal information based on the subject distance and the subject information acquired in step S1905. (S1906). In this embodiment, it is assumed that the normal image data and the template image data according to the type of subject information and the subject distance are retained.

Here, FIG. 20 shows the subject model normal information in the present embodiment as an example. As shown in FIG. 20, the subject model normal information includes subject model normal image data and subject model template image data. The subject model normal image data 2001, 2002, and 2003 shown in FIG. 20A are subject model normal image data corresponding to the subject distances of 0.5 m, 1.5 m, and 5.0 m, respectively.

Further, the subject model template image data 2004, 2005, 2006 shown in FIG. 20B is used to align the subject model normal image data 2001, 2002, 2003 with the color image data 501. In the present embodiment, it is assumed that the subject model template image data 2004, 2005, 2006 holds the contour information corresponding to the subject model normal image data 2001, 2002, 2003 as the image data.

In FIG. 20, as the subject model normal information, the subject model normal information corresponding to the face (characteristic pattern) of a person has been described as an example, but the image processing unit 208 similarly describes a dog, a cat, or the like. It is assumed that various subject model normal information is retained according to the subject distance.

The normal generation unit 304 generates normal image data based on the subject model normal information (S1907). In the generation of the normal image data, the normal generation unit 304 first uses the subject model template image data to set a parameter for aligning the subject model normal image data with respect to the subject of the color image data 501. calculate. In the present embodiment, the alignment parameter is calculated by extracting the contour information of the subject in the color image data 501 and matching it with the subject model template image data.

Next, the normal generation unit 304 converts the subject model normal image data according to the color image data 501 based on the calculated alignment parameters. Then, the normal generation unit 304 generates the normal image data by performing a smoothing process using the color image data as a reference image as in the first embodiment.

As described above, in the present invention, even if the subject distance changes, another image as if it was taken under desired lighting conditions can be generated from the captured image. In this embodiment, the subject information is acquired based on the user operation at the time of shooting acquired from the control unit 205 or the like (that is, the control unit 205 is made to function as the subject information acquisition means), but the subject information is acquired. The method is not necessarily limited to this. Therefore, for example, the subject information can be stored in the RAM 203 or the like as the tag information of the color image data 501, and the subject information can be acquired from the RAM 203. Further, it is also possible to separately provide a subject detection processing unit to detect a plurality of types of subjects such as faces, dogs, cats, etc., and acquire subject information based on this.

In addition, in the present embodiment, the contour information is retained as image data as the subject model template image data, but the texture and feature point information of the subject model may be retained. In this case, in step S1907, the alignment parameter may be calculated based on the texture and the feature point information.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

In addition, although the face of a person or the like has been mainly illustrated and described as a preferred embodiment in the above, the subject is not necessarily limited to the face of a person or the like. Therefore, for example, if an object has variations in normals in a predetermined portion and can be modeled for storing normal information, the object is applied as the subject of the present invention. can do.

105 Distance image acquisition unit 208 Image processing unit 301 Development processing unit 303 Face model normal information acquisition unit 304 Normal generation unit

Claims (12)

An image acquisition means for acquiring image data obtained by capturing a scene including a subject, and
The first acquisition means for acquiring the distance information from the image pickup device that captured the scene to the subject,
A storage means for storing normal information corresponding to a predetermined feature pattern related to the subject, and a storage means.
A generation means for generating normal information applied to a lighting process for the image data based on the distance information and the normal information stored in the storage means.
A lighting processing means for executing a lighting process on the image data by using the normal information generated by the generation means is provided .
The storage means stores a plurality of normal information corresponding to a predetermined feature pattern according to the distance from the image pickup apparatus to the subject.
The generating means is characterized in that the normal information applied to the lighting process is generated by using the normal information selected based on the distance information from the plurality of normal information stored by the storage means. Processing device. The first aspect of the present invention is characterized in that the generation means generates normal information to be applied to the lighting process by transforming the normal information selected based on the distance information according to the distance information. The image processing device described. The image processing device according to claim 1 or 2 , wherein the storage means stores the plurality of normal information at shorter distance intervals as the distance from the image pickup device to the subject is shorter. One of claims 1 to 3 , wherein the normal information stored by the storage means includes the normal information for each area set for each predetermined area in the predetermined feature pattern of the subject. The image processing apparatus described in the section. The image according to claim 4 , wherein the generation means generates normal information applied to the lighting process by aligning normal information for each area based on the distance information. Processing equipment. The generation means deforms the normal information for each region selected based on the distance information, and integrates the deformed normal information for each region, so that the normal information applied to the lighting process is applied. The image processing apparatus according to claim 4 , wherein the image processing apparatus is generated. Further provided with a second acquisition means for acquiring subject information indicating the type of the subject,
The image processing apparatus according to any one of claims 1 to 6 , wherein the generation means generates normal information corresponding to a predetermined feature pattern of the subject based on the subject information. The image processing apparatus according to claim 7 , further comprising a detection means for detecting the subject from the image data. The generation means aligns the detected subject with the normal information corresponding to a predetermined feature pattern of the subject selected based on the distance information, and thereby applies the normal to the lighting process. The image processing apparatus according to claim 8 , wherein information is generated. The image processing apparatus according to any one of claims 1 to 9 , wherein the predetermined feature pattern of the subject is a face. An image processing method in an image processing apparatus provided with a storage means for storing normal information corresponding to a predetermined feature pattern related to a subject.
An image acquisition step of acquiring image data obtained by capturing a scene including the subject by an image acquisition means, and an image acquisition step.
The first acquisition step of acquiring the distance information from the image pickup device that captured the scene to the subject by the first acquisition means, and
A generation step of generating normal information to be applied to the lighting process for the image data based on the distance information and the normal information stored by the storage means by the generation means.
The lighting processing means includes a lighting processing step of executing a lighting processing on the image data by using the normal information generated in the generation step.
The storage means stores a plurality of normal information corresponding to a predetermined feature pattern according to the distance from the image pickup apparatus to the subject.
The generating means is characterized in that the normal information applied to the lighting process is generated by using the normal information selected based on the distance information from the plurality of normal information stored by the storage means. Processing method. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 10.

Images