JP2019062436A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To generate an image taken under a desired illumination condition from a captured image even if a subject distance changes.SOLUTION: An image processing apparatus includes image acquisition means 301 for acquiring image data obtained by imaging a scene including a subject, first acquisition means 105 for acquiring distance information from an imaging apparatus that captures the scene to the subject, storage means 202 for storing normal line information corresponding to a predetermined feature pattern associated with the subject, generation means 304 for generating normal line information to be applied to lighting processing corresponding to the image data on the basis of the distance information and the normal line information stored in the storage means, and lighting processing means 306 for executing the lighting processing for the image data using the normal line information generated by the generation means.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In photographing a subject, the orientation, composition, lighting, etc. of the face are one of the major factors that determine the impression of the subject. Therefore, the subject is corrected by adjusting these shooting conditions after taking a picture. For example, Patent Document 1 discloses an image processing apparatus that corrects an object using a face model generated from a face solid shape template representing a face solid shape.

  In the image processing apparatus of Patent Document 1, the three-dimensional face shape template is deformed based on face information (that is, face position information, face size information, face parts information, and face direction information) detected from image data. The face model of the subject is generated. Further, in the image processing apparatus of Patent Document 1, a three-dimensional shape of a face is estimated based on the detected face part information, face size information, etc., and a three-dimensional shape template suitable for a subject from a plurality of face three-dimensional shape templates. Is selected. In this way, it is possible to select a three-dimensional face shape template in accordance with the difference in appearance due to the individual difference of the face, the orientation of the face, and the like.

JP, 2014-49866, A

  However, as in the image processing apparatus of Patent Document 1, when selecting the face solid shape template without considering at least the subject distance, when the subject distance changes, a face solid shape template different from the subject's appearance is selected. There is a possibility of As a result, the face model is also generated so as not to match the subject, and as a result, the subject corrected using the face model is generated as unnatural.

  The present invention has been made in view of the above-described conventional problems, and an object thereof is to generate an image as if taken under a desired illumination condition from a captured image even if the subject distance changes. It is.

  In order to achieve the above object, an image processing apparatus according to the present invention includes an image acquisition unit for acquiring image data obtained by imaging a scene including a subject, and a distance from the imaging apparatus which captured the scene to the subject A first acquisition unit for acquiring information, a storage unit for storing normal line information corresponding to a predetermined feature pattern related to the subject, the distance information, and the normal line information stored by the storage unit; A generation unit configured to generate normal information to be applied to the lighting processing on the image data; and a lighting processing unit configured to execute the lighting processing on the image data using the normal information generated by the generation unit. Prepare.

  According to the present invention, even if the subject distance changes, it is possible to generate an image as if taken under a desired illumination condition from the captured image.

It is a figure showing the appearance of the imaging device provided with the image processing device. It is a block diagram showing an internal configuration of an imaging device provided with an image processing device. It is a block diagram showing functional composition of an image processing device. 5 is a flowchart showing the procedure of processing in the image processing apparatus. It is a figure which shows color image data and distance image data. It is a figure which shows a face area | region and an organ position. It is a flowchart which shows the procedure of a face model normal line information acquisition process. It is a figure which shows face model normal information. It is a flowchart which shows the procedure of normal image data generation processing. It is a figure which shows the outline | summary of normal image data generation processing. It is a figure which shows the outline | summary of a lighting process. It is a figure which shows the result of a lighting process. It is a flowchart which shows the procedure of a face model normal line information acquisition process. It is a figure which shows area classified face model normal line information. It is a figure which shows face model normal information. It is a flow chart which shows a procedure of field model face information normal information acquisition processing according to field. It is a figure which shows the outline | summary of a face normal line image data generation process. It is a block diagram showing functional composition of an image processing device. 5 is a flowchart showing the procedure of processing in the image processing apparatus. It is a figure showing subject model normal line information.

(Appearance of the imaging device)
FIG. 1 is a view showing an appearance of an imaging apparatus provided with an image processing apparatus according to the embodiment, and FIG. 1A shows an appearance of a front surface of the imaging apparatus and FIG. . The imaging apparatus 101 includes an optical unit 102, an imaging button 103, a flash 104, a distance image acquisition unit 105, a display unit 106, and an operation button 107.

  The optical unit 102 is a lens barrel configured of a zoom lens, a focus lens, a shake correction lens, an aperture, and a shutter, and collects light information of an object. The imaging button 103 is a button for the user to instruct the imaging apparatus 101 to start imaging. The strobe 104 is an illumination that can be emitted in response to the start of imaging according to a user's instruction.

  The distance image acquisition unit 105 acquires, as distance information, distance image data of an object in a scene captured according to an imaging instruction. Here, the distance image data is image data in which the subject distance corresponding to the pixel is stored as the pixel value of each pixel of the image. The distance image acquiring unit 105 includes an infrared light emitting unit that emits infrared light, and a light receiving unit that receives infrared light reflected by the subject. The distance image acquisition unit 105 calculates the distance value from the imaging device to the subject based on the time until the infrared light emitted from the infrared light emitting unit is reflected on the subject and the light receiving unit receives the light. Then, based on the calculated distance value and distance imaging information including the number of sensor pixels of the light receiving unit, the angle of view, etc., the position information of the object is calculated, and distance image data is generated. In addition, the acquisition method of distance image data is not necessarily restricted to this. Therefore, for example, an optical system similar to the optical unit 102 is provided instead of the distance image acquisition unit 105, and triangulation is performed based on the parallax between the image data captured from two different viewpoints. You can also get it.

  The display unit 106 is a display device such as a liquid crystal display that displays image data processed by the imaging device 101 and other various data. Note that as shown in FIG. 1, since the imaging device 101 is not provided with an optical finder, a framing operation (confirmation of focus and composition) is performed using the display unit 106. That is, since the imaging is performed while confirming the live view image in the display unit 106 regarding the imaging device 101, the display unit 106 also functions as an electronic finder when the framing and focusing operations are performed. In addition, the display unit 106 also displays a camera setting menu and the like.

  The operation button 107 is a button for the user to instruct the imaging device 101 to switch the operation mode of the imaging device 101, various parameters at the time of imaging, and the like. Note that the imaging apparatus 101 includes, as one of the operation modes, a lighting processing mode in which the degree of illumination of the captured image is corrected after imaging. Therefore, the user uses the operation button 107 or the imaging button 103 to switch to the lighting processing mode, sets the illumination parameters of the virtual illumination used for the lighting processing, and selects the subject whose illumination condition is adjusted (corrected), etc. It can be performed. Further, when outputting the corrected image data, the user can also instruct whether or not to output the distance image data. In addition, the display unit 106 may have a touch screen function. In that case, it is also possible to handle a user instruction using the touch screen as an input of the operation button 107.

(Internal configuration of imaging device)
FIG. 2 is a block diagram showing an internal configuration of an imaging apparatus provided with the image processing apparatus according to the embodiment. The CPU 201 relates to all the processing of each configuration, and executes processing by sequentially reading and interpreting instructions stored in a ROM (Read Only Memory) 202 and a RAM (Random Access Memory) 203.

  The ROM 202 stores a program executed by the CPU 201. In the present embodiment, the ROM 202 stores a plurality of face model normal line information and area-specific face model normal line information. The face model normal information corresponds to a face of a predetermined shape (face model), face model normal image data in which a normal vector of the face surface is stored in pixel values, and an eye and a nose in the face model normal image data -Including a face model normal organ position indicating the position of an organ such as the mouth. In addition, region-specific face model normal information includes region-specific face model normal image data in which normal vectors corresponding to regions such as face contour, eyes, nose, and mouth are stored in pixel values, and region-specific face model normals. And a region-specific face model normal organ position indicating an organ position in the image data. The RAM 203 stores a program to be executed by the CPU 201.

  The optical system control unit 204 is a control circuit that performs control instructed by the CPU 201 such as focusing on the optical unit 102, opening the shutter, adjusting the aperture, and the like. In addition, the optical system control unit 204 can acquire the distance to the focused object based on the state of the optical system and the like, and store the distance in the RAM 203 or the like.

  A control unit 205 is a control circuit that receives a user instruction from the imaging button 103 or the operation button 107, controls switching to an imaging mode, a lighting processing mode, etc., selects a subject region, sets illumination parameters, and the like. The color imaging element unit 206 is an imaging element that converts the light information collected by the optical unit 102 into a current value. The color imaging element unit 206 includes a color filter having a predetermined arrangement such as a Bayer arrangement, and acquires color information of an object from light collected by the optical unit 102.

  The A / D conversion unit 207 is a processing circuit that converts color information of the subject detected in the color imaging device unit 206 into digital signal values to obtain RAW image data. In the present embodiment, it is assumed that distance image data and RAW image data captured at the same time can be acquired. The image processing unit 208 performs development processing on the RAW image data acquired by the A / D conversion unit 207, and generates color image data. Further, the image processing unit 208 performs various image processing such as generation of corrected image data obtained by performing lighting processing on color image data using color image data and distance image data. The internal configuration and functions of the image processing unit 208 will be described in detail with reference to FIG. 3 described later.

  The character generation unit 209 is a processing circuit that generates characters, graphics, and the like. The characters and graphics generated by the character generation unit 209 are displayed on the display unit 106 so as to be superimposed on image data, corrected image data, and the like. The encoder unit 210 converts various image data including color image data processed by the image processing unit 208 and corrected image data generated by lighting processing into a file format such as Jpeg.

  The media I / F 211 is an interface for transmitting and receiving image data to and from the PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.). As the media I / F 211, for example, USB (Universal Serial Bus) or the like is used. The system bus 212 is a bus for transmitting and receiving data.

(Internal configuration of image processing unit)
FIG. 3 is a block diagram showing the functional configuration of the image processing apparatus (image processing unit 208) according to the embodiment. The development processing unit 301 performs white balance processing, demosaicing processing, noise reduction processing, color conversion processing, and the like on the RAW image data acquired from the A / D conversion unit 207, and generates color image data. The imaging apparatus 101 can output color image data generated by the development processing unit 301 to the display unit 106 for display, or can store the color image data in a storage device such as the RAM 203 or the PC / media 213.

  The face detection unit 302 acquires face information of a subject from color image data acquired from the development processing unit 301. The face information of the subject includes at least information on a face area indicating an area occupied by the face of the subject in the color image data and an organ position indicating a position in the color image data such as eyes and mouth included in the face. .

  The face model normal information acquisition unit 303 selects normal line information matching the subject from the plurality of pieces of face model normal line information based on the distance information to the subject. In the present embodiment, face model normal line information is selected from the ROM 202 based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105.

  The normal generation unit 304 acquires face information from the face detection unit 302, face model normal information from the face model normal information acquisition unit 303, and distance image data from the distance image acquisition unit 105, respectively, based on the information. And generates normal image data corresponding to the color image data.

  The illumination parameter setting unit 305 sets an illumination parameter based on a user operation (instruction) received from the control unit 205. The illumination parameters in the present embodiment include the position, attitude, intensity, light source color, and the like of the illumination. The lighting processing unit 306 acquires the distance image data from the distance image acquisition unit 105, the normal image data from the normal generation unit 304, and the illumination parameter from the illumination parameter setting unit 305, respectively, and a color image is obtained based on those data. Perform a writing process on the data. The imaging apparatus 101 can output the corrected image data corrected by the lighting processing to a storage device such as the RAM 203 or the PC / media 213 for storage or output to the display unit 106 for display.

(Processing flow of image processing unit)
FIG. 4 is a flowchart showing the procedure of processing in the image processing apparatus (image processing unit 208) according to the embodiment. In the process illustrated in the flowchart of FIG. 4, the image processing unit 208 first selects face model normal line information based on face information and distance image data acquired from color image data. Next, the image processing unit 208 generates normal image data corresponding to color image data based on the face information, the face model normal information, and the distance image data. The image processing unit 208 further executes lighting processing for adding virtual illumination to color image data based on the illumination parameters, distance image data and normal image data set by the user, and generates corrected image data.

  The processing procedure of the image processing unit 208 will be described in detail below. The development processing unit 301 generates color image data based on the RAW image data acquired from the A / D conversion unit 207 (S401). The color image data is shown, for example, as shown in FIG. 5A, and RGB values are stored as pixel values in the pixel I (i, j) of the color image data 501, and each of the Ir (i, j) j) shown as Ig (i, j), Ib (i, j). The method of acquiring color image data is not necessarily limited to this. Therefore, for example, RAW image data stored in the RAM 203 or PC / media 213 can be acquired, and the development processing unit 301 can generate color image data from the acquired RAW image data. Further, color image data stored in the RAM 203 or the PC / media 213 can also be obtained directly.

  The image processing unit 208 acquires distance image data from the distance image acquisition unit 105 (S402). The distance image data is shown, for example, as shown in FIG. 5B, and the pixel D (i, j) of the distance image data 502 stores the distance value from the imaging device to the subject. In addition, the acquisition method of distance image data is not necessarily restricted to this. Therefore, for example, the distance image data stored in the RAM 203 or the PC / media 213 can also be obtained directly.

  Next, the image processing unit 208 acquires the state of the flag SW1 from the RAM 203, and determines whether to execute the writing process based on the state of the flag SW1 (S403). When the flag SW1 is OFF, the image processing unit 208 determines that the writing process is not to be performed (S403 No), and the process proceeds to Step S404. Further, when the flag SW1 is ON, it is determined that the lining process is to be performed (S403: Yes), and the process proceeds to Step S405.

  If it is determined in step S403 that the writing process is not to be performed, the image processing unit 208 acquires the state of the flag SW2 from the RAM 203. Then, the image processing unit 208 outputs only color image data when the flag SW2 is OFF, and outputs color image data and distance image data to the PC / media 213 when the flag SW2 is ON (S404). Alternatively, the image processing unit 208 can output color image data to the display unit 106 and can display the color image data so that the user can confirm. When the image processing unit 208 executes the processing in step S404, the processing illustrated in FIG. 4 ends.

  If it is determined in step S403 that the lighting process is to be performed, the face detection unit 302 detects (acquires) face information based on the color image data (S405). Here, the face information will be described with reference to FIG. As shown in FIG. 6, the face information includes a face area 601 and an organ position 602. The face area is indicated in the image data as a set of pixels of the area including the face. The organ position 602 is shown as coordinates corresponding to the eyes, nose, mouth, etc. in the face area.

  An existing algorithm can be applied to a method of detecting a face area and an organ position. The existing algorithms include an algorithm using template matching, an algorithm using Haar-Like feature quantities, and the like. In this embodiment, the face region / organ position is detected by applying template matching.

  In the detection to which template matching is applied, first, a skin color area is extracted as a face candidate area by performing threshold processing on color image data. Next, matching processing is performed on the face candidate area using templates of face images of various sizes to calculate likelihood. Then, based on the calculated likelihood, a process of determining whether or not it is a face area is executed to extract a face area. Also for the organ position, the matching process is similarly performed on the extracted face area using the template of the eye, nose, and mouth image, and the likelihood is calculated. Then, based on the magnitude of the likelihood, the reliability of the face area / organ position is calculated. The face area 601 and the organ position 602 are acquired by executing the above processing.

  Next, the face model normal information acquisition unit 303 acquires face model normal information based on the subject distance (S406). The process of acquiring face model normal information in the face model normal information acquisition unit 303 will be described in detail with reference to FIG.

  The normal line generation unit 304 corresponds to the color image data generated in step S401 based on the distance image data, face information, and face model normal information acquired in steps S402, S405, and S406, respectively. Line image data is generated (S407). The process of generating the normal line image data in the normal line generation unit 304 will be described in detail with reference to FIGS. 9 and 10 described later.

  The illumination parameter setting unit 305 sets (determines) illumination parameters based on the user operation received from the control unit 205 (S408). The lighting processing unit 306 executes lighting processing such as adding virtual illumination (virtual light source) to color image data with respect to color image data based on distance image data, normal image data, and illumination parameters. The correction image data is generated (S409). The writing process in the writing process unit 306 will be described in detail with reference to FIG.

  The lighting processing unit 306 determines whether to end the lighting processing based on the user operation (that is, an instruction from the user) received from the control unit 205 (S410). If it is determined that the lighting processing is to be ended by the lighting processing unit 306, the image processing unit 208 shifts the processing to step S411, and if it is determined that the lighting processing is not ended, the processing returns to step S408. Set

  When the process proceeds to step S411, the image processing unit 208 acquires the state of the flag SW2 from the RAM 203, and the lighting processing unit 306 outputs various data based on the acquired state of the flag SW2. The lighting processing unit 306 outputs the color image data and the correction image data when the flag SW2 is OFF, and outputs the color image data, the correction image data, and the distance image data to the PC / media 213 when the flag SW2 is ON. Remember. Thereafter, the image processing unit 208 ends the processing illustrated in FIG.

(Face model normal information acquisition processing)
Here, in step S406, the process of acquiring face model normal information (that is, the process of acquiring face model normal information based on distance image data and face information) executed by the face model normal information acquisition unit 303. explain.

  Hereinafter, the procedure of the face model normal information acquisition process will be described in detail with reference to FIG. The face model normal information acquisition unit 303 first calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105 (S701). In the present embodiment, the average of the distance values in the face area (that is, the area occupied by the face of the subject) is taken as the subject distance.

  The face model normal information acquisition unit 303 acquires face model normal information based on the subject distance calculated in step S701 (S702). In the present embodiment, face model normal information closest to the subject distance calculated in step S701 is selected from a plurality of face model normal information corresponding to the subject distance shown in FIG.

  FIG. 8 illustrates a plurality of face model normal line information according to the subject distance. In FIG. 8, face model normal information 801, 802, 803 are face model normal information according to subject distances of 0.5 m, 1.5 m, and 5.0 m, respectively, and these face model normal information is It is stored in the ROM 202.

  The face model normal information includes face model normal image data storing face vector normal vectors (that is, the direction of the face surface when observed from the position of the object distance) in pixel values, and face model normal image data A face model normal organ position indicating an organ position such as eyes, nose and mouth in Note that pixel values indicating background pixels are stored in pixels (that is, background pixels) in areas other than the face area in the face model normal image data.

  As shown in FIG. 8, in the case of a frontal face, organ positions such as eyes and mouth move away from the center as the subject distance decreases (that is, approach to the face contour) and organ positions such as eyes and mouth as the subject distance increases. Looks like it was in the middle of the face. As described above, the balance between the face contour and the eyes, nose, mouth, and the like changes according to the subject distance, and the change in organ position increases as the subject distance decreases. Therefore, with regard to face model normal information, it is preferable to hold face model normal information at short distance intervals as the subject distance is shorter.

  Further, in FIG. 8, the case where the face model normal line information is three has been described as an example, but the number of face model normal line information is not limited to this. Therefore, for example, depending on the capacity of the ROM 202 or the like, the number of pieces of face model normal line information may be set to two or four or more. If the number of pieces of face model normal information is set to a large number, it is possible to select face model normal information that conforms to the conditions at the time of shooting.

(Normal image data generation process)
Here, normal image data generation processing performed by the normal generation unit 304 in step S407 (ie, a normal image corresponding to color image data based on face information, face model normal information, and distance image data A process of generating data will be described. Hereinafter, the procedure and outline of the normal image data generation process will be described in detail with reference to FIGS. 9 and 10, respectively.

  The normal generation unit 304 generates subject normal image data based on the distance image data (S901). Regarding generation of subject normal line image data according to the present embodiment, the normal line generation unit 304 first generates a two-dimensional coordinate value of a pixel in the distance image data and a distance value acquired from the pixel value of the pixel. , Three-dimensional coordinate values are calculated for each pixel. Then, the normal-line generating unit 304 next calculates a normal vector corresponding to each pixel based on the calculated coordinate value. Note that the normal vector corresponding to the pixel of interest is calculated based on the gradient calculated from the three-dimensional coordinate values of the vicinity region of the pixel of interest. Thus, the normal line generation unit 304 generates the subject normal line image data 1001.

  Next, the normal generation unit 304 generates face normal image data based on the face information and the face model normal information (S902). Regarding generation of face normal image data according to the present embodiment, the normal generation unit 304 generates a face model based on the organ position 602 included in the face information and the face model normal organ position included in the face model normal information. The face normal image data 1002 is generated by deforming the normal image data. In the present embodiment, affine transformation is used to deform face model normal image data so as to fit the face of the subject in the color image data 501. Specifically, an affine parameter is calculated from the organ position 602 and the face model normal organ position using the least squares method, and the face model normal image parameter is deformed based on the calculated affine parameter. Line image data 1002 is generated. In the face normal image data, background pixels other than the face store pixel values indicating the background pixels.

  When the subject normal image data 1001 and the face normal image data 1002 are generated, the normal generation unit 304 integrates them and smoothes them to generate normal image data (S903). Specifically, first, the pixels in the area other than the face area of the face normal image data 1002 (that is, background pixels) are the pixel values of the subject normal image data 1001, and the other pixels are the face normal image data 1002. The image data is integrated so as to be a pixel value. That is, integrated normal image data 1003 is generated. Next, the normal line generation unit 304 generates normal line image data 1004 by performing smoothing processing on the generated integrated normal line image data 1003. In the present embodiment, smoothing processing is applied to the integrated normal image data 1003 using the color image data 501 as a reference image. In this case, for example, a joint bilateral filter using the color image data 501 as a reference image may be applied.

  By processing as described above, normal image data can be generated. In addition, the process which produces | generates normal line image data is not necessarily limited to the above-mentioned process. Therefore, for example, normal image data can be generated only from the face normal image data 1002 without generating the object normal image data 1001. Further, in this case, instead of the object normal image data 1001, image data in which appropriate normal vectors are stored in all pixels may be provided.

(Lighting process)
Here, the lighting processing performed by the lighting processing unit 306 in step S409 will be described. As described above, the lighting process in step S409 adds virtual illumination to color image data based on distance image data and normal image data and further according to the illumination parameters set by the user, and corrects the corrected image. It is processing to generate. In the present embodiment, the corrected image I ′ generated by the lighting process is generated, for example, according to the following equation (1).

  In equation (1), I'r, I'g, I'b are the pixel values of the corrected image data I ', Lrm, Lgm, Lbm are the colors of the m-th illumination, and km is the degree of correction for the m-th illumination Represent. Furthermore, the correction degree km for the mth illumination is determined based on the position Q of the virtual illumination, the intensity (brightness of illumination) α, the posture U, the distance value corresponding to the pixel (x, y), and the normal vector N For example, it is calculated by the following equation (2). The color L of the virtual illumination, the position Q, the intensity α, and the attitude U are illumination parameters set by the user. Further, t is a correction coefficient for adjusting the degree of correction by the virtual illumination, and is set to 1 in the present embodiment.

  Hereinafter, the description of the equation (2) will be supplemented with reference to FIG. In FIG. 11, Q indicates the position of the virtual illumination as described above, and P indicates the three-dimensional position of the pixel (i, j). Note that P is the distance value from the imaging device 101 to each pixel (that is, the distance value of each pixel) acquired based on the pixel value of the distance image data 502, the angle of view of the imaging device 101, color image data 501 It is calculated based on the image size of W is a function that returns a larger value as the distance from the position P of the pixel (i, j) to the position Q of the virtual illumination increases. That is, the function W corrects the pixel (i, j) to be brighter as it is closer to the virtual illumination.

  In equation (2), K is a function that returns a larger value as ρ is smaller, and ρ is a vector V (i, j) from the position P of the pixel (i, j) to the position Q of the virtual illumination , It is shown by the angle which the attitude | position U of illumination makes. Further, N (i, j) is a normal vector corresponding to the pixel (i, j). In addition, since the inner product of the vector V (i, j) and the normal vector N is included in the numerator of the equation (2), the vector V from the pixel P (i, j) toward the virtual illumination and the normal vector N It is corrected that the smaller the angle formed, the brighter the pixel. By performing this correction, for example, as shown in FIG. 12 and the like, the color image data 501 can be corrected to a corrected image 1201 as if the subject was illuminated by virtual illumination. That is, as shown in the above equation, by performing the lighting process (by generating a correction image), it is possible to correct the brightness according to the position of the illumination and the shape of the subject.

  As described above, according to the present invention, even if the subject distance changes, it is possible to generate another image as if taken under a desired illumination condition from the captured image. Specifically, even when the subject is of the same size on the photographed image, the subject is photographed at a long subject distance using a lens having a long focal length, and the subject is short using a lens having a short focal length The lighting process produces different effects depending on the distance taken.

  In the present embodiment, the face normal information is described as a specification to be acquired from the ROM 202 based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105. The data acquisition method is not necessarily limited to this. Therefore, for example, face normal information can be selected in step S702 based on the subject distance acquired when the optical system control unit 204 brings the subject into focus in step S701. If the RAW image data stored in the RAM 203 or the PC / media 213 includes the subject distance as tag information, the subject distance is acquired in step S701, and the face normal information is input in step S702. It can also be selected.

Second Embodiment
In the first embodiment, the face model normal information suitable for the subject is selected from the plurality of face model normal information in accordance with the subject distance, but in the present embodiment (the second embodiment), in accordance with the subject distance. Face model normal information is generated from region-specific face model normal information. Thereby, face model normal line information corresponding to various subject distances can be acquired.

  FIG. 14 shows region-specific face model normal line information as an example. The area-specific face model normal line information includes the area-specific face model normal image data 1401 to 1405 shown in FIG. 14A and the area-specific face model normal organ position shown in FIG.

  Region-specific face model normal image data 1401 to 1405 shown in FIG. 14A respectively represent normal vectors of the face surface in the face outline shape, the right eye peripheral region, the left eye peripheral region, the nose peripheral region, and the mouth peripheral region. Image data stored in pixel values.

  In addition, as shown in FIG. 14B, in the region-specific face model normal image data 1402-1405 corresponding to the right eye peripheral region, left eye peripheral region, nose peripheral region, and mouth peripheral region, organ reference An organ reference position indicating the position to be set is set. That is, the region-specific face model normal organ position indicates the position of the organ corresponding to the subject distance in the region-specific face model normal image data 1401 corresponding to the face outline shape. In the example shown in FIG. 14, it is assumed that the region-specific face model normal organ position corresponding to the subject distance of 0.5 m, 1.5 m, and 5.0 m is held (ie, the area for each subject distance). Another face model normal organ position is stored).

  In the second embodiment, the process of acquiring face model normal information in step S406 is different from that of the first embodiment. FIG. 13 is a flowchart showing a procedure of acquisition processing of face model normal information in the image processing apparatus (image processing unit 208) according to the second embodiment of the present invention.

  Hereinafter, the procedure of face model normal information acquisition processing will be described in detail with reference to FIG. The face model normal information acquisition unit 303 calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105, as in step S701 of the first embodiment. (S1301).

  When the face model normal information acquisition unit 303 calculates the subject distance, the face model organ position corresponding to the subject distance acquired in step S1301 is determined based on the region model face model normal organ position shown in FIG. It is calculated by interpolation processing (S1302).

  Here, the description will be supplemented regarding the calculation of the face model organ position. Assuming that the acquired subject distance is l, an area corresponding to the subject distance lA which is smaller than l which is the acquired subject distance and is closest to l among the subject distances for which the region-specific face model normal organ position is set. The coordinates (xA, yA) of another face model normal organ position are selected. Similarly, an area-specific face model normal organ position corresponding to an object distance lB that is larger than the acquired object distance l and is closest to l among the object distances for which the area-specific face model normal organ position is set. Select the coordinates (xB, yB) of. And in this case, face model organ position (x ', y') can be calculated | required like the following Formula (3).

  In Equation (3), w is a weight calculated from l, 1A, and 1B, and can be obtained, for example, as the following Equation (4).

  In Equation (4), γ is a parameter for adjusting the weight w, and is an interval (for example, 0.1 m, 0.5 m, 1.0 m) of the subject distance in which the position of the region-specific face model normal organ is set. Etc.) can be adjusted. Regarding γ, the subject distance interval for which the region-specific face model normal organ position is set is set to 1 if the interval is shorter than a predetermined value, and if the interval is longer than the predetermined value Is set to a value greater than one. By setting in this way, it is possible to increase the movement amount of the organ as the subject distance is shorter and to decrease the movement amount of the organ as the subject distance is longer.

  The face model normal information acquisition unit 303 generates face model normal image data based on the region-specific face model normal image data and the calculated face model organ position (S1303). Specifically, with respect to region-specific face model normal image data 1401 corresponding to the face outline shape, other region-specific face model normal image data 1402-1402 are matched so that the face model organ position and the organ reference position coincide. Align 1405. Then, the face model normal image data is generated by overwriting the pixel value of the pixel having normal information of the region-specific face model normal image data 1402 to 1405 with respect to the region-specific face model normal image data 1401. Do. FIG. 15 shows face model normal image data 1501 generated by the processing shown in FIG. 13 as an example thereof.

  As described above, according to the present invention, even if the subject distance changes, it is possible to generate another image as if taken under a desired illumination condition from the captured image. In the present embodiment, although it has been described that one region-by-region face model normal image data is held for each region as region-by-region face model normal information, it corresponds to different object distances. A plurality of region-specific face model normal image data may be held. Therefore, for example, region-specific face model normal image data corresponding to object distances of 0.5 m, 1.5 m, and 5.0 m are held, and the object distance closest to the object distance calculated in step S1301 is stored. Area-specific face model normal image data can also be acquired. In addition, it is possible to change the number of face model normal image data classified by area, which is held for each area (organ). For example, a plurality of face model normal image data corresponding to the face outline shape and the nose peripheral area are held, and the face model normal image data corresponding to the right eye peripheral area, the left eye peripheral area and the mouth peripheral area is one. Only the sheet may be held.

Third Embodiment
In the second embodiment, the face model normal information is generated based on the region-specific face model normal information. However, in the present embodiment (third embodiment), the region-specific face model normal is generated according to the subject distance. Information is selected to generate face normal image data.

  In the third embodiment, step S406 is different in the relationship (comparison) with the first embodiment, and instead of the processing for acquiring face model normal information in step S406, processing for acquiring area model face normal information is executed. . Further, the processing content of the face normal image data generation processing of step S902 is different from that of the first embodiment. Hereinafter, these processes (that is, the process of acquiring face model normal information by region and the process of generating face normal image data) will be described in detail.

(Area-specific face model normal information acquisition processing)
Here, an area-specific face model normal information acquisition process executed by the face model normal information acquisition unit 303 will be described. In this embodiment, area-specific face model normal line information is acquired based on distance image data and face information in regard to the process of acquiring area-specific face model normal line information. FIG. 16 shows the procedure of region type face model normal information acquisition processing.

  The face model normal information acquisition unit 303 calculates the subject distance based on the face information acquired from the face detection unit 302 and the distance image data acquired from the distance image acquisition unit 105, as in step S701 of the first embodiment. (S1601).

  The face model normal information acquisition unit 303 selects region model face normal information corresponding to the acquired subject distance (S1602). Note that, in the processing of step S1602, region-specific face model normal line information closest to the acquired subject distance is selected. For example, in the case of region-by-region face model normal information shown in FIG. 14, the region-by-region face model normal image data 1401 to 1405 and the region-by-region face model normal image position are the closest to the acquired subject distance To get

(Face normal image data generation process)
Here, the face normal image data generation process executed by the normal line generation unit 304 will be described. In the present embodiment, face normal image data is obtained based on the region-specific face model normal information (that is, the region-specific face model normal image data 1401 to 1405 and the region-specific face model normal organ position) acquired in step S1602. Generate FIG. 17 shows an outline of face normal image data generation processing.

  First, using the affine transformation, the normal line generation unit 304 deforms the region-specific face model normal image data so as to fit the face of the subject in the color image data 501. The affine parameters of the region-specific face model normal image data 1401 corresponding to the face outline shape are calculated using the least squares method based on the organ position 602 and the region face model normal organ position selected in step S1602. Ru.

  Next, the affine parameters of other regional face model normal image data 1402-1405 are translated so that the organ reference position corresponding to the organ position 602 and the regional face model normal image data 1402-1405 match. Calculated as corrected components. Thereby, the region-specific face model normal image data 1402 to 1405 can be deformed in accordance with the organ position 602 of the object of the color image data 501.

  Then, the normal-line generating unit 304 deforms the region-by-region face model normal image data 1401 to 1405 using the conversion parameter calculated as described above, and generates region-by-region face normal image data 1701 to 1705. Furthermore, the normal line generation unit 304 generates face normal image data 1706 by integrating the region-by-area face normal image data 1701 to 1705.

  That is, in the present embodiment, the pixels having normal line information in the region-by-region face normal image data 1702 to 1705 other than the face general shape are overwritten on the region-by-region face normal image data 1701 corresponding to the face general shape. , Face normal image data 1706 is generated. Note that the pixels serving as the background in all of the region-by-region face normal image data 1702 to 1705 (that is, the pixels having no normal line information related to the region) are region-by-region face normal images corresponding to the face outline shape. Face normal image data 1706 is generated using the pixel values of the data 1701.

  As described above, according to the present invention, even if the subject distance changes, it is possible to generate another image as if taken under a desired illumination condition from the captured image. In the present embodiment, the region model normal model position closest to the acquired subject distance is selected in step S1602, but as described above in step S1302 in the second embodiment, the region is obtained by interpolation processing. Another face model normal organ position may be calculated.

  Further, in the present embodiment, it has been described that one region-by-region face model normal image data is held for each region as region-by-region face model normal information, but region-by-region face model normals corresponding to different subject distances A plurality of image data may be held. Therefore, for example, region-specific face model normal image data corresponding to subject distances of 0.5 m, 1.5 m, and 5.0 m are held, and the subject distance closest to the subject distance calculated in step S 1601 is corresponded. Area-specific face model normal image data can also be acquired.

  In addition, it is also possible to change the number of region-specific face model normal image data held for each region (organ). For example, a plurality of face model normal image data corresponding to the face outline shape and the nose peripheral area are held, and the face model normal image data corresponding to the right eye peripheral area, the left eye peripheral area and the mouth peripheral area is one. Only the sheet may be held.

Fourth Embodiment
In the first to third embodiments, the processing target is mainly described as a person, but in the present embodiment, in addition to the person (more specifically, the face of the person), other subjects are also processed. . This makes it possible to acquire normal line information according to the subject distance for various subjects.

(Internal configuration of image processing unit)
FIG. 18 is a block diagram showing a functional configuration of the image processing apparatus (image processing unit 208) according to the embodiment (the fourth embodiment) of the present invention. In the functional configuration of the image processing unit 208 shown in FIG. 18, subject model normal information instead of the face model normal information acquisition unit 303 in relation to the image processing apparatus (image processing unit 208) according to the first embodiment. An acquisition unit 1801 is implemented. In addition, the processing content of the normal generation unit 304 is also different.

  The subject model normal line information acquisition unit 1801 selects normal line information matching the subject from among a plurality of subject model normal line information, based on the distance information to the subject and the subject information. In the present embodiment, subject information selected by a user operation is acquired from the control unit 205. The subject information indicates the type of the subject included in the color image data, and can be selected from, for example, dogs, cats and the like in addition to the person. Further, in this case, it is possible to set only a dog, a cat, etc., which is a subject other than a person, as the subject of the subject.

(Processing flow of image processing unit)
FIG. 19 is a flowchart showing the procedure of processing in the image processing apparatus (image processing unit 208) according to the embodiment of the present invention. The flowchart shown in FIG. 19 differs from the flowchart shown in FIG. 4 in the processes (steps S1905 to S1907).

  Hereinafter, each process of step S1905 to step S1907 will be described. If it is determined in step S1903 in FIG. 19 that the lighting process is to be performed, the subject model normal information acquisition unit 1801 acquires subject information based on a user operation from the control unit 205 (S1905). The subject model normal information acquisition unit 1801 acquires subject model normal line information corresponding to the subject in the color image data 501 from among a plurality of subject model normal line information based on the subject distance and the subject information acquired in step S1905. (S1906). In the present embodiment, it is assumed that normal image data and template image data are held according to the type of subject information and subject distance.

  Here, FIG. 20 shows subject model normal line information in this embodiment as an example. The subject model normal information includes subject model normal image data and subject model template image data, as shown in FIG. Subject model normal image data 2001, 2002, and 2003 shown in FIG. 20A are subject model normal image data according to subject distances of 0.5 m, 1.5 m, and 5.0 m, respectively.

  Further, subject model template image data 2004, 2005, 2006 shown in FIG. 20B is used to align subject model normal image data 2001, 2002, 2003 with color image data 501. In the present embodiment, it is assumed that contour information corresponding to subject model normal image data 2001, 2002, 2003 is held as image data as subject model template image data 2004, 2005, 2006.

  In FIG. 20, subject model normal line information corresponding to a person's face (feature pattern) has been described as an example of the subject model normal line information. It is assumed that various subject model normal line information is held according to the subject distance.

  The normal line generation unit 304 generates normal line image data based on the subject model normal line information (S1907). In generating normal image data, the normal-line generating unit 304 first uses the subject model template image data to set parameters for aligning subject model normal image data with respect to the subject in the color image data 501. calculate. In the present embodiment, outline information of a subject in color image data 501 is extracted, and matching with subject model template image data is performed to calculate alignment parameters.

  Next, the normal-line generation unit 304 converts the subject model normal image data into the color image data 501 based on the calculated alignment parameter. Then, the normal line generation unit 304 generates normal line image data by performing smoothing processing using color image data as a reference image as in the first embodiment.

  As described above, according to the present invention, even if the subject distance changes, it is possible to generate another image as if taken under a desired illumination condition from the captured image. In the present embodiment, subject information is acquired based on the user operation at the time of shooting acquired from the control unit 205 or the like (that is, although the control unit 205 functions as a subject information acquisition unit), acquisition of subject information The method is not necessarily limited to this. Therefore, for example, subject information may be stored in the RAM 203 or the like as tag information of the color image data 501, and the subject information may be acquired from the RAM 203. Alternatively, a subject detection processing unit may be separately provided to detect a plurality of types of subjects such as a face, a dog, and a cat, and subject information may be acquired based on the detected subjects.

  In addition, in the present embodiment, contour information is held as image data as subject model template image data, but textures and feature point information of the subject model may be held. In this case, in step S1907, alignment parameters may be calculated based on the texture and feature point information.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  Besides, in the above, as a preferred embodiment, mainly the face and the like of a person are illustrated and described, but the subject is not necessarily limited to the face and the like of a person. Therefore, for example, if it is an object having a variation in normal in a predetermined part, and the object can be modeled in storing normal information, the object is applied as the subject of the present invention can do.

105 distance image acquisition unit 208 image processing unit 301 development processing unit 303 face model normal information acquisition unit 304 normal generation unit

Claims (13)

An image acquisition unit that acquires image data obtained by imaging a scene including a subject;
A first acquisition unit that acquires distance information from the imaging device that has captured the scene to the subject;
Storage means for storing normal line information corresponding to a predetermined feature pattern associated with the subject;
Generation means for generating normal line information to be applied to lighting processing on the image data based on the distance information and the normal line information stored in the storage means;
An image processing apparatus comprising: lighting processing means for performing a lighting process on the image data using the normal line information generated by the generation means. The storage means stores a plurality of normal line information corresponding to a predetermined feature pattern in accordance with the distance from the imaging device to the subject,
The image processing apparatus according to claim 1, wherein the generation unit selects normal line information from a plurality of normal line information stored in the storage unit based on the distance information.   3. The apparatus according to claim 2, wherein the generation means generates normal line information to be applied to the lighting process by deforming normal line information selected based on the distance information according to the distance information. Image processing apparatus as described.   4. The image processing apparatus according to claim 2, wherein the storage unit stores the plurality of pieces of normal line information at shorter distance intervals as the distance from the imaging device to the subject is shorter.   The normal line information stored in the storage means includes normal line information for each area set for each predetermined area in a predetermined feature pattern of the subject. An image processing apparatus according to claim 1.   6. The image according to claim 5, wherein the generation means generates normal line information to be applied to the lighting process by aligning the normal line information for each area based on the distance information. Processing unit.   The generation means generates normal information to be applied to the lighting process by transforming the normal information by region selected based on the distance information according to the distance information and further integrating the information. The image processing apparatus according to claim 5, wherein It further comprises a second acquisition unit for acquiring subject information indicating the type of the subject,
The image processing apparatus according to any one of claims 1 to 7, wherein the generation unit generates normal line information corresponding to a predetermined feature pattern of the subject based on the subject information.   9. The image processing apparatus according to claim 8, further comprising detection means for detecting the subject from the image data.   The normal means applied to the lighting process by aligning the detected subject with normal information corresponding to a predetermined feature pattern of the subject selected based on the distance information. The image processing apparatus according to claim 9, generating information.   The image processing apparatus according to any one of claims 1 to 10, wherein the predetermined feature pattern of the subject is a face. An image processing method in an image processing apparatus, comprising storage means for storing normal line information corresponding to a predetermined feature pattern related to a subject,
An image acquisition step of acquiring image data obtained by imaging a scene including the subject by the image acquisition means;
A first acquisition step of acquiring distance information from the imaging device that has captured the scene to the subject by the first acquisition unit;
A generation step of generating normal information to be applied to lighting processing on the image data based on the distance information and the normal information stored by the storage unit;
And a lighting processing step of performing lighting processing on the image data using the normal line information generated in the generation step by a lighting processing means.   A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11.