JP2019101978A - Image processing device, image processing method, and program - Google Patents

Abstract

To prevent the reproduction of an unnatural shadow, when generating an image in which a shadow under a virtual illumination condition is reproduced on the basis of the three-dimensional shape data of a subject.SOLUTION: An image processing device relating to one embodiment of this invention which adds a shadow effect by virtual illumination to a captured image on the basis of three-dimensional shape data corresponding to the captured image includes extraction means for a subject area which extracts the three-dimensional shape data of the subject area from the three-dimensional shape data and movable range setting means which sets the movable range of the virtual illumination on the basis of the loss of the three-dimensional shape data, when the three-dimensional shape data of the subject area has a loss.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for adding a shadow effect as if captured under a desired illumination condition to a captured image.

  When the subject is photographed by the imaging device, the photographed image largely changes depending on how the illumination (light) strikes the subject. For example, under an illumination condition in which light is obliquely applied to a subject that is a person, the shadow of the face is enhanced, and a captured image of a three-dimensional impression can be obtained. However, under an illumination condition such as strong backlighting where light is coming from behind the subject, a part or the whole of the subject is shaded, resulting in a dark captured image. Thus, Patent Documents 1 and 2 disclose methods of adding a shading effect to a captured image as if captured under illumination conditions desired by the user. Patent Document 1 discloses a method of using a three-dimensional shape data of an object to generate an image that reproduces a shadow under virtual illumination conditions and combining the image with a captured image. Further, Patent Document 2 discloses a method of setting the movable range of the illumination position of the virtual illumination in accordance with the size of the subject, the radius of curvature, and the like.

  In the following, as in Patent Documents 1 and 2, processing for generating an image that reproduces a shadow under virtual illumination conditions based on three-dimensional shape data of an object after shooting is referred to as lighting processing.

JP, 2013-125292, A Unexamined-Japanese-Patent No. 2015-201839

  Patent Documents 1 and 2 above disclose that lighting processing is performed based on three-dimensional shape data of a subject. However, the three-dimensional shape data of the subject does not necessarily represent the accurate three-dimensional shape of the subject. For example, the three-dimensional shape data may not acquire the three-dimensional shape of the entire circumference of the subject, and a defect may exist in the three-dimensional shape. In addition, the accuracy of obtaining three-dimensional shape data may be low, and there may be unevenness that is not originally present on the surface of the subject. As described above, the three-dimensional shape data may not accurately represent the original three-dimensional shape of the subject. When lighting processing is performed based on such three-dimensional shape data, unnatural shadows are reproduced. There are times when

  The present invention has been made in view of such problems, and an object thereof is to generate an image that reproduces a shadow under virtual illumination conditions based on three-dimensional shape data of an object. In order to prevent unnatural shadows from being reproduced.

  An image processing apparatus according to an embodiment of the present invention is an image processing apparatus that adds a shading effect by virtual illumination to the captured image based on three-dimensional shape data corresponding to the captured image, and the three-dimensional shape data From the object region extraction means for extracting three-dimensional shape data of the object region, and when there is a defect in the three-dimensional shape data of the object region, the virtual is based on the loss of the three-dimensional shape data. And moving range setting means for setting the moving range of the illumination.

  In addition, an image processing apparatus according to an embodiment of the present invention is an image processing apparatus that adds a shading effect by virtual illumination to the captured image based on three-dimensional shape data corresponding to the captured image, Based on the acquisition accuracy of the three-dimensional shape data, when the acquisition accuracy of the three-dimensional shape data of the object region is low, the extraction unit of the object region extracts the three-dimensional shape data of the object region from shape data. And illumination size range setting means for setting the settable range of the illumination size of the virtual illumination.

  According to the present invention, it is possible to prevent unnatural shadows from being reproduced when generating an image that reproduces shadows under virtual illumination conditions based on three-dimensional shape data of a subject.

FIG. 1 is a view showing an appearance of an image processing apparatus according to a first embodiment. FIG. 1 is a diagram showing an internal configuration of an imaging device according to Embodiment 1. FIG. 2 is a functional block diagram showing the configuration of an image processing unit according to the first embodiment. FIG. 7 is a diagram showing parameters representing the illumination position of virtual illumination according to the first embodiment. 5 is a flowchart showing the flow of image processing according to the first embodiment. FIG. 5 is a diagram for explaining loss of three-dimensional shape data according to the first embodiment. FIG. 6 is a diagram showing a method of setting a movable range of the illumination position according to the first embodiment. FIG. 6 is a view showing a display example of a movable range of the illumination position according to the first embodiment. FIG. 7 is a view showing a display example of a warning for the illumination position according to Embodiment 1. FIG. 2 is a functional block diagram showing a configuration of a lighting processing unit according to Embodiment 1. 5 is a flowchart showing a flow of processing of a lighting processing unit according to the first embodiment. FIG. 5 is a diagram showing an example of shadow image data according to the first embodiment. FIG. 6 is a diagram showing an example of lighting image data according to the first embodiment. FIG. 6 is a diagram for explaining the effect according to the first embodiment. FIG. 7 is a functional block diagram showing the configuration of an image processing unit according to a second embodiment. 7 is a flowchart showing the flow of processing according to the second embodiment. FIG. 7 is a view showing a method of setting a movable range of the illumination position according to the second embodiment. FIG. 13 is a functional block diagram showing the configuration of an image processing unit according to a third embodiment. FIG. 14 is a view for explaining setting operation of illumination conditions and parameters according to the third embodiment. 15 is a flowchart showing the flow of image processing according to the third embodiment. FIG. 18 is a schematic view illustrating normals of shape data according to the third embodiment. FIG. 14 is a diagram for explaining a method of setting a settable range of the illumination size according to the third embodiment. FIG. 14 is a view showing a display example of a settable range of the illumination size according to the third embodiment. FIG. 14 is a view showing a display example of a warning for the illumination size according to the third embodiment. FIG. 13 is a functional block diagram showing the configuration of a lighting processing unit according to a third embodiment. It is a flowchart which shows the flow of a process of the lighting process part which concerns on Embodiment 3. FIG. FIG. 14 is a diagram showing a configuration of an imaging system according to a fourth embodiment. FIG. 16 is a diagram showing an internal configuration of an imaging device according to a fourth embodiment. FIG. 18 is a functional block diagram showing the configuration of an image processing unit according to a fourth embodiment. 15 is a flowchart illustrating the flow of image processing according to a fourth embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Embodiment 1
<Appearance of Imaging Device>
FIG. 1 is a view showing the appearance of an image processing apparatus applicable to the present embodiment. In the present embodiment, a digital camera that is an imaging device is shown as an example of the image processing device, but the image processing device is not limited to this. For example, the image processing apparatus may be an information processing apparatus such as a mobile phone, a tablet device, a personal computer or the like, or may be configured as an imaging apparatus such as a camera-equipped mobile phone.

  FIG. 1A shows the appearance of the front of the image processing apparatus according to this embodiment, and FIG. 1B shows the appearance of the back. An imaging apparatus 101 illustrated as an example of an image processing apparatus includes a color image imaging unit 102, an imaging button 103, a distance image acquisition unit 104, a display unit 105, and an operation button 106.

  The color image capturing unit 102 includes a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, and a color imaging device unit, and acquires light information of a subject. The configuration of the color image capturing unit 102 is not limited to this. The imaging button 103 is a button for the user to instruct the imaging apparatus 101 to start imaging. The distance image acquisition unit 104 acquires distance image data in which the distance to the subject is recorded according to the imaging instruction from the user. In the present embodiment, the color image pickup unit 102 and the distance image acquisition unit 104 are included in the housing of the image pickup apparatus 101, but the present invention is not limited to this. The color image pickup unit 102 and the distance image acquisition unit 104 may be an independent configuration. Further, the distance image acquiring unit 104 in the present embodiment acquires the distance in the optical axis direction of the distance image acquiring unit 104 based on the distance image acquiring unit 104, and a color image capturing unit based on the color image capturing unit 102. Convert to a distance in the optical axis direction of 102 and save. The position of the distance image acquisition unit 104 and the color image pickup unit 102 in advance and the optical axis direction are obtained and calibrated to obtain distance image data based on the distance image acquisition unit 104 as a color image pickup unit. It can be converted into distance image data based on 102. Hereinafter, the distance image data D acquired by the distance image acquisition unit 104 is expressed by the pixel D (i, j). Further, it is assumed that each pixel of the distance image data D is associated with each pixel of the color image data I. Specifically, it is assumed that the pixels I (i, j) of the color image data I correspond to the pixels D (i, j) of the distance image data D on a one-to-one basis. Any existing method may be used as a method used to acquire the distance based on the distance image acquisition unit 104 in the distance image acquisition unit 104. For example, a time-of-flight method that calculates the distance from the time until the emitted light is reflected back to the subject, a multi-view stereo method that calculates the distance by triangulation from images taken from two or more multiple locations, etc. Can be used.

  The display unit 105 is configured of a liquid crystal display or the like, and can display a live view image at the time of shooting by the imaging device 101, and can display a captured image after shooting. Further, in the present embodiment, the display unit 105 displays the preview image of the lighting process when the user designates the position of the virtual illumination on the captured image, and the display of the processing result image after the lighting process is performed. Do. The display unit 105 can also display a setting menu of the imaging apparatus 101.

  The operation button 106 is a button for the imaging device 101 to acquire a user operation. Specifically, the operation button 106 is a button for the user to instruct the imaging apparatus 101 to switch the imaging mode or set various parameters at the time of imaging with respect to the setting menu displayed by the display unit 105. . The user can use the operation button 106 or the imaging button 103 to instruct the imaging device 101 to set illumination parameters, select a subject area, and the like. Note that the display unit 105 may have a touch screen function, and the imaging device 101 can also handle a user instruction via the touch screen as an input of the operation button 106.

<Internal Configuration of Imaging Device>
FIG. 2 is a block diagram showing an internal configuration of the imaging apparatus 101 in the present embodiment.

  The CPU 202 sequentially reads and interprets instructions stored in a ROM (Read Only Memory) 203 and a RAM (Rondom Access Memory) 204, and executes processing of each configuration according to the result. That is, an instruction is a computer program. The system bus 212 is a bus for transmitting and receiving data.

  The optical system control unit 205 performs, for the color image capturing unit 102, control instructed by the CPU 202 such as focusing, opening a shutter, adjusting an aperture, and the like.

  The control unit 206 controls imaging, selection of an object area, setting of an illumination position of virtual illumination, and the like according to user instructions from the imaging button 103 and the operation button 106. When setting the illumination position of the virtual illumination, the control unit 206 performs an illumination position update mode indicating which parameter is to be updated among a plurality of parameters indicating the illumination position described later, and an illumination position indicating whether the illumination position is determined Control the decision flag. In the illumination position update mode, among a plurality of parameters representing the illumination position, 0 is set to update the azimuth and elevation angle of the illumination position, and 1 is set to update the radius of the illumination position. The illumination position determination flag is set to 0 when the illumination position is undecided and 1 when the illumination position is decided. Both the illumination position update mode and the illumination position determination flag are given 0 as an initial value.

  A character generation unit 207 generates characters, graphics, and the like. The generated characters and graphics are superimposed on the lighting image generated by the image processing unit 209 and displayed on the display unit 105.

  The A / D conversion unit 208 converts the light amount of the subject detected by the color image capturing unit 102 into a digital signal value to generate RAW image data. In the present embodiment, it is assumed that distance image data and RAW image data captured at the same time can be acquired.

  The image processing unit 209 performs various types of image processing on color image data generated by performing development processing on the above-described RAW image data. In the present embodiment, the image processing unit 209 performs lighting processing based on color image data and distance image data, and generates lighting image data from the color image data. The internal structure of the image processing unit 209 will be described in detail later.

  The encoder unit 210 performs conversion processing of converting various image data including color image data processed by the image processing unit 209 and lighting image data generated by the lighting processing into a file format such as Jpeg.

  The media I / F 211 is an interface for transmitting and receiving image data to and from the PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.).

<Configuration of Image Processing Unit 209>
FIG. 3 is a functional block diagram showing the configuration of the image processing unit 209 in the present embodiment. In the following, details of functions for the image processing unit 209 to perform the lighting process in the present embodiment will be described.

  The image processing unit 209 receives raw image data from the A / D conversion unit 208, distance image data from the distance image acquisition unit 104, and an illumination position of virtual illumination from the control unit 206. The image processing unit 209 performs lighting processing on the RAW image data based on the distance image data and the illumination position. Then, the image processing unit 209 outputs an output image, which is the result of the writing process, to the PC / media 213 (from the encoder unit 210 via the media I / F 211) or to the display unit 105.

  Each functional block of the image processing unit 209 will be described below.

  The image acquisition unit 301 acquires RAW image data from the A / D conversion unit 208. The developing unit 302 performs demosaicing processing on the RAW image data to generate color image data. The generated color image data can be output to the display unit 105 for display or can be stored in a storage device such as the RAM 204 or the PC / media 213. In the present embodiment, it is assumed that RGB values are stored as pixel values in the pixels I (i, j) constituting the color image data I. The RGB values stored in each pixel of the color image data are denoted as Ir (i, j), Ig (i, j), Ib (i, j). Further, hereinafter, the number of pixels in the horizontal direction of color image data is denoted as W, and the number of pixels in the vertical direction of color image data is denoted as H.

  The shape acquisition unit 303 acquires three-dimensional shape data of a photographed scene (hereinafter, also simply referred to as a scene) including a subject. In the present embodiment, three-dimensional shape data is obtained by converting the distance image data acquired by the distance image acquisition unit 104 into three-dimensional shape data. The three-dimensional shape data in the present embodiment is also referred to as a polygon, and is data consisting of a surface formed by connecting a large number of vertices in a three-dimensional space and a plurality of vertices such as 3 or 4. The n-th (n ≦ N, N is the number of vertices) vertices in the polygon hold three-dimensional coordinates Vn = (Xn, Yn, Zn) representing positions in the three-dimensional space. The format of the three-dimensional shape data is not limited, and other formats may be used.

  The illumination position setting unit 304 receives a user operation for changing the illumination position input from the control unit 206, and sets the illumination position of the virtual illumination in accordance with an instruction from the user. In the present embodiment, color image data is displayed on the display unit 105 having a touch screen function, and the illumination position of the virtual illumination is received according to an instruction from the user obtained by a drag operation on the display screen. As shown in FIG. 4, the illumination position of the virtual illumination in this embodiment is the center position of rotation C = (Cx, Cy, Cz), the rotation radius r around the center position C, the rotation angle θ around the Y axis This is expressed using four variables of the rotation angle φ around the Z axis. Hereinafter, the rotation angle θ around the Y axis is referred to as an azimuth angle, and the rotation angle φ around the Z axis is referred to as an elevation angle. Using these four variables C, r, θ, and φ, the three-dimensional coordinate L = (Lx, Ly, Lz) of the illumination position can be obtained by the following equation (1).

  Note that the method of expressing the illumination position is not limited to this, and may be represented in another form.

  The imaging device position / posture calculation unit 305 calculates the position and orientation of the imaging device 101 when the color image data is captured from the color image data and the three-dimensional shape data. Here, the position of the imaging device 101 represents the three-dimensional position of the optical center of the imaging device 101. Further, the attitude of the imaging device 101 represents the direction of the optical axis of the imaging device 101. The position and orientation of the imaging device 101 may be calculated by calculating the relative positional relationship between the subject and the imaging device 101.

  The lighting processing unit 306 performs lighting processing and color conversion processing using color image data, three-dimensional shape data, the position / posture of the imaging device 101, and the illumination position of the virtual illumination. The lighting processing unit 306 adds shading effect to color image data (that is, a captured image) by lighting processing to generate lighting image data. Furthermore, the lighting processing unit 306 performs color conversion processing such as white balance processing and gamma processing on the generated lighting image data to generate final output image data.

  The image output unit 307 outputs the output image data received from the writing processing unit 306 to the PC / media 213 via the display unit 105 or the encoder unit 210.

  The subject region extraction unit 308 specifies the region of the subject of interest among the three-dimensional shape data from the color image data and the three-dimensional shape data, and extracts the three-dimensional shape data of the subject region. Hereinafter, the area of the subject of interest in the three-dimensional shape data is simply referred to as a subject area. Further, an area other than the target subject in the three-dimensional shape data is referred to as a background area.

  The shape determination unit 309 determines whether there is a defect in the three-dimensional shape data of the subject area. Details of the process will be described later.

  The illumination position movable range setting unit 310 sets the movable range of the illumination position of the virtual illumination based on the loss of the three-dimensional shape data of the subject region when it is determined that the loss exists in the three-dimensional shape data of the subject region. . Specifically, the movable range is set for each parameter representing the illumination position of the virtual illumination. In the present embodiment, the movable range is set for the azimuth angle θ and the elevation angle φ among the parameters representing the illumination position. The movable range may be set with respect to the central position C and the radius r among the parameters representing the illumination position.

  The illumination position movable range output unit 311 outputs the movable range of the illumination position of the virtual illumination set by the illumination position movable range setting unit 310 to the display unit 105 and presents it to the user.

  The illumination position evaluation unit 312 evaluates whether the user-specified illumination position set by the illumination position setting unit 304 is included in the movable range of the illumination position set by the illumination position movable range setting unit 310.

  The warning output unit 313 outputs a warning to the display unit 105 when the lighting position evaluation unit 312 evaluates that the lighting position designated by the user is not included in the movable range.

  As described above, the image processing unit 209 in the present embodiment has a function for performing the lighting process.

<Processing Flow of Image Processing Unit 209>
Subsequently, the processing flow of the lighting processing by each unit of the image processing unit 209 in the present embodiment will be described. FIG. 5 shows a flowchart of the lighting process in the present embodiment. The processing of each step in the flowchart is implemented by the CPU 202 reading out and executing a program stored in the ROM 203 or the RAM 204.

  First, in step S501, the image acquisition unit 301 acquires RAW image data captured by the imaging apparatus 101 from the A / D conversion unit 208.

  Next, in step S502, the developing unit 302 develops the RAW image data to generate color image data. Generally, development processing of RAW image data includes various processing such as white balance adjustment, gamma correction, noise removal and the like including demosaicing, but in the present embodiment, only demosaicing and noise removal are performed. It is desirable that the RGB values Ir (i, j), Ig (i, j), Ib (i, j) of each pixel of the color image data store a value proportional to the radiance received by the pixel. .

  Next, in step S503, the shape acquisition unit 303 acquires polygons that are three-dimensional shape data of a photographed scene including a subject. In the present embodiment, the shape acquiring unit 303 generates polygons by using, as vertices, three-dimensional points corresponding to respective pixels of the distance image data acquired by the distance image acquiring unit 104 and connecting the vertices as surfaces. . The details of the process performed by the shape acquisition unit 303 will be described below.

  First, the shape acquisition unit 303 calculates the position of the three-dimensional point Vn corresponding to each pixel of the distance image data D. Specifically, from the pixel at coordinates (i, j) in the distance image data D, the position Vn = (Xn, Yn, Zn) of the three-dimensional point is calculated by the following equation (2).

  Here, cxd and cyd are values representing an optical center called a principal point of the distance image acquisition unit 104 by coordinates in the distance image. fd is a value representing the focal length of the distance image acquisition unit 104 in pixel conversion of distance image data. W represents the number of pixels in the horizontal direction of the distance image data. The three-dimensional position obtained by the above equation (1) takes an optical center of the color image pickup unit 102 of the image pickup apparatus 101 as an origin, a direction horizontal to the image pickup apparatus X axis, and a direction perpendicular to the image pickup apparatus Y axis 3 represents a position in a three-dimensional space in which the optical axis direction of the imaging device is taken as the Z axis. Note that n and (i, j) may be in one-to-one correspondence, and n and (i, j) may be related by an equation other than the above equation. Moreover, although Formula (1) is a formula which calculates position Vn of a three-dimensional point from two-dimensional coordinates (i, j) of distance image data D, the position of a three-dimensional point is solved by solving this formula in reverse. The two-dimensional coordinates (i, j) of the distance image data D corresponding to Vn can also be calculated.

  Subsequently, the shape acquisition unit 303 connects a plurality of three-dimensional points to form a surface, thereby generating a polygon. In this embodiment, in order to generate three-dimensional points from distance image data arranged in a grid pattern at equal intervals, a plane can be configured by connecting adjacent points on this grid. Further, it is also possible to connect three-dimensional points to form a surface by using an existing method such as Delaunay triangulation.

  Next, in step S504, the imaging device position and orientation calculation unit 305 calculates relative position and orientation data of the imaging device when capturing color image data from the three-dimensional shape data and the color image data. In the present embodiment, position / orientation data of the imaging device are represented by three variables representing the three-dimensional coordinates of the optical center of the imaging device, the horizontal direction of the imaging device, the vertical direction of the imaging device, and the optical axis direction of the imaging device. I assume.

  Next, in step S505, the subject region extraction unit 308 extracts three-dimensional shape data of the subject region from the color image data and the three-dimensional shape data. In the present embodiment, the subject area extraction unit 308 first extracts the subject of interest in the color image data I, and thereafter obtains the area corresponding to the subject of interest of the color image data I from the three-dimensional shape data. A known method may be used as a method of extracting a subject of interest in the color image data I. For example, a subject of interest may be extracted automatically by using an image recognition technology. In this case, for example, when the subject is a person, person recognition technology or face recognition technology can be used. In addition, the control unit 206 may manually select a region of interest to the user. A set of two-dimensional coordinates extracted as a subject area in the color image data I is taken as OI. In the present embodiment, as described above, the color image data I and the distance image data D correspond to pixels at the same coordinates. Therefore, in the color image data I, the coordinate group OI representing the subject area also represents the subject area in the distance image data D. By using the equation (1), it is possible to calculate a three-dimensional point Vn corresponding to each coordinate of the two-dimensional coordinate group OI among the three-dimensional shape data. In the following, it is assumed that a point included in the subject region in the three-dimensional shape data is O. Further, the three-dimensional position OC of the center of gravity of the subject region of the three-dimensional shape data can be calculated by the following equation (3).

  Here, | O | represents the number of points included in the set O.

Next, in step S506, the illumination position movable range setting unit 310 initializes the movable range of the illumination position of the virtual illumination. With respect to the illumination position, the illumination position movable range setting unit 310 sets the rotation center position C as the three-dimensional position OC of the center of gravity of the subject region described above, and is a parameter representing the illumination position: center position C, radius r, azimuth angle θ The movable range is set for the azimuth angle θ and the elevation angle φ of the elevation angle φ. In the present embodiment, the movable range of each of the azimuth angle θ and the elevation angle φ is expressed by the following equation (4) using θ ₁ , θ ₂ , φ ₁ and φ ₂ .

As the initial value of the movable range, a value represented by the following equation (5) is set to each of θ ₁ and θ ₂ .

Similarly, initial values are set to φ ₁ and φ ₂ as shown in the following equation (6).

  Next, in step S507, the shape determination unit 309 determines whether there is a defect in the three-dimensional shape data of the subject region. In the present embodiment, the loss of three-dimensional shape data indicates a state in which three-dimensional shape data can not be acquired all around the subject. When the subject shape is acquired from only one direction as in the distance image acquisition unit 104, the three-dimensional shape data of the front of the subject can be acquired, but the three-dimensional shape data of the back can not be acquired. Therefore, since the three-dimensional shape data of the entire circumference of the subject can not be acquired, the shape determination unit 309 determines that there is a defect in the three-dimensional shape data of the subject. FIG. 6 is a diagram for explaining loss of three-dimensional shape data in the present embodiment. As shown in FIG. 6A, a case of capturing a scene consisting of a subject 601 and a background area 602 other than the subject using the imaging device 101 will be described as an example for the loss of three-dimensional shape data. Although the case of photographing a person as the subject 601 will be described, the subject is not limited to the person. FIG. 6B is a view of the imaging apparatus 101, the subject 601, and the background area 602 as viewed from above. The portions visible from the distance image acquisition unit 104 and capable of acquiring distance data are indicated by solid lines 603 and 605, and the portions invisible from the distance image acquisition unit 104 and impossible to acquire distance data are indicated by dotted lines 604 and 606. . As described above, with respect to a portion where the distance data represented by dotted lines 604 and 606 can not be acquired, the three-dimensional shape data of the subject is lost. FIG. 6C shows an example 610 of color image data in this case.

  In the present step S507, when there is a defect in the three-dimensional shape data of the subject area, the process proceeds to step S508. On the other hand, if it is determined that no defect exists in the three-dimensional shape data, the process proceeds to step S510.

  The presence or absence of loss of three-dimensional shape data can be determined as follows. For example, the shape determination unit 309 can determine the presence or absence of a loss by determining whether a polygon forming the subject region is an open surface or a closed surface. If the polygon is an open surface, the shape determination unit 309 determines that a defect exists. In some cases, the correct three-dimensional shape data (i.e., a polygon) may be an open surface. Therefore, for example, it is also possible to combine the application results of existing image recognition technology to color image data. That is, it is possible to know whether the original three-dimensional shape data is an open curved surface or an open curved surface by determining what kind of subject the image recognition technology is. Furthermore, although the three-dimensional shape data should originally be a closed surface, if the acquired three-dimensional shape data is an open surface, it is determined that a defect exists in the acquired three-dimensional shape data. Further, the predicted result of the three-dimensional shape data of the portion where the shape data does not exist may be compared with the acquired three-dimensional shape data, and if there is a difference, it may be determined whether the open surface is a defect. As a method of predicting three-dimensional shape data of a portion where shape data does not exist, when it is possible to recognize what kind of subject it is by the above-mentioned image recognition technology, average shape data of the subject is used as original three-dimensional shape data There is a way to use it. Besides, the radius of curvature of the area in which the three-dimensional shape data can be acquired may be calculated, and the three-dimensional shape data may be complemented by the curvature. Further, as another method of determining the presence or absence of a defect, the determination may be made uniformly depending on the device used for acquiring the three-dimensional shape data of the subject. For example, when using a device that acquires three-dimensional shape data from only one direction as in the distance image acquisition unit 104, it may be determined that a defect exists in the three-dimensional shape data. On the other hand, in the case of using a multi-viewpoint camera in which the subject is surrounded from the entire circumference, it may be determined that no defect exists in the three-dimensional shape data. The method of determining the presence or absence of a defect is not limited, and other methods may be used.

Next, in step S508, when there is a defect in the three-dimensional shape data of the subject, the illumination position movable range setting unit 310 updates the illumination position movable range based on the subject region of the three-dimensional shape data. In the present embodiment, first, a range of an area in which shape data can be acquired among three-dimensional shape data of a subject is determined. A method of determining the range of the subject shape in the present embodiment will be described with reference to FIG. FIG. 7 is a schematic view of the imaging device 101 and the subject 701 as viewed from above, as in FIG. 6B. It should be noted that there is a defect in the three-dimensional shape data of the subject 701, and FIG. 7 shows only the area where the three-dimensional shape data can be acquired. In FIG. 7, reference numeral 702 denotes the center of gravity OC of the subject region of the three-dimensional shape data calculated in step S505. A line segment 703 is a line connecting the imaging device 101 and the center of gravity OC of the subject region. Points 704 and 705 are three-dimensional points corresponding to the boundary between the subject area and the non-subject area in the three-dimensional shape data. As a method of obtaining these points, first, pixels forming an outline of the subject are extracted from the subject region of the color image data I in step S505. Then, a three-dimensional point corresponding to the outline pixel of the color image data I can be obtained by back-projecting the two-dimensional coordinates to the three-dimensional shape data by using the equation (1). In FIG. 7, a line segment 706 is a line segment connecting the center of gravity 702 and the point 704. Similarly, a line segment 707 is a line segment connecting the center of gravity 702 and the point 705. Then, it calculates the angle theta ₁ which the line segment 703 and line segment 706 forms the two angles between the angle theta ₂ which the line segment 703 and line segment 707 forms. The two angles theta ₁ and theta _2, in the present embodiment deals with a range of subject shape. In FIG. 7, the angles (azimuth angles) θ ₁ and θ ₂ are shown as a range of the subject shape in two dimensions, but in fact, the subject 701 has three-dimensional shape data. Therefore, angles (elevation angles) φ ₁ and φ ₂ are similarly calculated as the range of the subject shape when viewed from the side of the subject.

Then, the movable range with respect to the illumination position of a new virtual illumination is calculated based on the range of the area | region which can acquire the above-mentioned shape data. As shown in the following equation (7), the movable ranges (azimuth angles) θ ₁ and θ ₂ are updated using the above-mentioned angles θ ₁ and θ ₂ and the variable α.

Similarly, the movable ranges (elevation angles) φ ₁ and φ ₂ are updated as shown in the following equation (8) using the angles φ ₁ and φ ₂ and the variable β.

  Here, the variables α and β are variables taking real numbers in the range of 0 <α, β ≦ 1, and these values may be set in advance, or may be set by the user of the imaging apparatus 101. Good. Thus, the illumination position movable range setting unit 310 updates the illumination position movable range based on the subject region of the three-dimensional shape data when there is a defect in the three-dimensional shape data of the subject.

Next, in step S 509, the illumination position movable range output unit 311 outputs the movable range of the illumination position and displays the movable range on the display unit 105. The display of the movable range of the illumination position is performed using figures and characters. For example, an area corresponding to the movable range of the illumination position can be displayed by hatching with respect to the color image data I captured by the imaging unit. Conversely, an area outside the movable range of the illumination position may be shaded. FIG. 8 shows a display example of the movable range of the illumination position on the display unit 105. First, the display position and the size of the movable range of the illumination position will be described with reference to FIG. In FIG. 8A, the inside of the boundary 801 of the movable range of the illumination position is the movable range of the illumination position. The boundary line 801 is displayed centered on the point 802. A point 802 is coordinates of color image data corresponding to the barycentric position OC of the three-dimensional shape data of the subject area described above. The point 802 can be calculated from the three-dimensional coordinates of the center-of-gravity position OC based on the equation (1). A point 803 represents the position on the color image data of the illumination position at θ = θ ₁ and φ = 0 ° centering on the gravity center position OC of the object region of the three-dimensional shape data. The radius r can be set to any value, and may be set in advance or may be set by the user. A point 804 represents a position on color image data corresponding to a three-dimensional position represented by θ = θ ₂ and φ = 0 ° at an arbitrary radius r around the barycentric position OC. Points 803 and 804 indicate the movable range of the illumination position in the horizontal direction. Similarly, a point 805 represents a position on color image data corresponding to a three-dimensional position represented by θ = 0 ° and φ = φ ₁ at an arbitrary radius r around the barycentric position OC. A point 806 represents a position on color image data corresponding to a three-dimensional position represented by θ = 0 ° and φ = φ ₂ at an arbitrary radius r around the barycentric position OC. Point 805 and point 806 represent the movable range of the illumination position in the vertical direction. A curve connecting these four points 803, 804, 805, and 806 is a boundary 801 of the movable range of the illumination position. FIG. 8B illustrates an example of a screen when the user sets the illumination position 809 of the lighting process on the subject 807 of color image data. In this screen, the boundary line 801 of the movable range of the illumination position 809 whose position and size are determined as described above is displayed superimposed on the color image data. A message 808 indicates that the setting of the illumination position for performing the lighting process is in progress. The user moves the illumination position 809 by a drag operation or the like to set the illumination position 809. In the present embodiment, the movable range of the illumination position 809 is indicated during setting of the illumination position 809. Note that by shading the area outside the movable range of the illumination position, it may be visually shown that the illumination position can not be set in that area. In addition, the movable range of the illumination position may be displayed as characters 810 in the display unit 105.

  Next, in step S510, the illumination position setting unit 304 sets an initial value for each of the illumination rotation center position C, radius r, azimuth angle θ, and elevation angle φ as initial values of the illumination position. Set As the rotation center position C of the illumination, the center of gravity OC of the subject region of the three-dimensional shape data is set. As for the radius r, the azimuth angle θ, and the elevation angle φ, any position can be set within the movable range of the illumination position previously determined. For example, the position of illumination that illuminates the subject from the front is set as an initial value. For that purpose, the value of the Z component of the center of gravity OC of the object region is set to the radius r, and 0 ° is set to each of the azimuth angle θ and the elevation angle φ. The Z component of the center of gravity OC of the subject area represents the distance from the imaging apparatus 101 to the subject.

  Next, in step S511, the lighting processing unit 306 performs lighting processing using color image data, three-dimensional shape data, imaging device position / posture data, and illumination position, and further performs color conversion processing to output Generate image data. Details of the writing process will be described later.

  Next, in step S512, the image output unit 307 outputs the output image data to the display unit 105, and performs preview display of the lighting processing result. The user looks at the lighting processing result displayed as a preview, and if it is determined that the desired shade is reproduced, the user depresses the operation button 106 to determine the lighting position. At this time, the control unit 206 updates the illumination position determination flag to one.

  Next, in step S513, the image output unit 307 determines whether the illumination position has been determined. Specifically, the image output unit 307 refers to the illumination position determination flag, and if the value is 1 representing determination, the process proceeds to step S514, and if 0 indicating undetermined, the process proceeds to step S515.

  In step S514, the image output unit 307 outputs the output image data to the PC / media 213 for storage.

In step S515, the illumination position setting unit 304 updates the illumination position in accordance with the user's drag operation. When the illumination position update mode is 0, the azimuth θ and the elevation angle φ of the illumination position are updated. Assuming that the pressing start coordinate on the color image data of the drag operation by the user is P _s = (i _s , j _s ) and the pressing end coordinate is P _e = (i _e , j _e ), θ using the following equation (9) 'And φ' are calculated, and substituted into azimuth angle θ and elevation angle φ, respectively, and updated.

  Here, A and B are represented by the following formula (10), where u is the angle of view in the horizontal direction of the imaging apparatus 101 and v is the angle of view in the vertical direction. As described above, W indicates the horizontal direction of color image data, and H indicates the number of pixels in the vertical direction of color image data.

On the other hand, when the illumination position update mode is 1, the radius r of the illumination position is updated. Here, a new radius r is calculated using the amount of movement in the horizontal direction of the drag operation by the user. Specifically, the radius r ′ is calculated from the pressing start coordinate P _s of the drag operation and the pressing end coordinate P _e using the following equation (11), and this is substituted into the radius r and updated.

Next, in step S516, the illumination position evaluation unit 312 determines whether the illumination position designated by the user is included in the above-described illumination position movable range. Here, the azimuth angle θ representing the illumination position and the elevation angle φ are compared with the illumination position movable range to determine whether the illumination position designated by the user is included in the movable range. The azimuth angle θ is compared with the angles θ ₁ and θ ₂ indicating the illumination position movable range. The elevation angle φ is compared with the angles φ ₁ and φ ₂ representing the illumination position movable range. If both the azimuth angle θ and the elevation angle φ are included in the illumination position movable range, the process proceeds to step S511. On the other hand, when one of the azimuth angle θ and the elevation angle φ is not included in the illumination position movable range, the process proceeds to step S517.

  In step S517, the warning output unit 313 outputs a message (that is, character information) indicating that the illumination position specified by the user is not included in the illumination position movable range or a warning including a figure to the display unit 105, and displays it. Do. FIG. 9 shows an example of a warning display in the present embodiment. In FIG. 9, an icon 901 representing the illumination position is set outside the illumination position movable range. When the illumination position is thus set outside the movable range, the warning output unit 313 displays a warning message 902. In addition to displaying the warning message 902, the color of the icon 901 representing the illumination position may be changed. When changing the color of the icon 901, the color is switched between two types of illumination position movable range inside and outside of the illumination position movable range, and even within the illumination position movable range, when approaching the boundary of the illumination position movable range The color may be changed. The warning display method is not limited to these.

  As described above, the image processing unit 209 in the present embodiment performs the lighting process. By this processing, a lighting image in which a virtual lighting effect is added to the photographed color image data is generated.

<Operation of Lighting Processing Unit 306>
The operation of the lighting processing unit 306 in the present embodiment will be described in detail below. As described above, the lighting processing unit 306 performs lighting processing using color image data, three-dimensional shape data, imaging device position / posture data, and illumination position, and further performs color conversion processing. Generate output image data.

  FIG. 10 shows the configuration of the lighting processing unit 306 in the present embodiment. The lighting processing unit 306 includes a normal calculation unit 1001, a shadow image generation unit 1002, an image combining unit 1003, and a color conversion processing unit 1004. The normal line calculation unit 1001 calculates normal line data representing the direction of the surface of the shape from the three-dimensional shape data (i.e., a polygon). A shadow image generation unit 1002 generates shadow image data by calculating a shadow when the scene is illuminated with virtual illumination from the three-dimensional shape data, the normal data, the illumination position, and the imaging device position / posture data. . The image combining unit 1003 combines the shadow image data and the color image data output from the developing unit 302 to generate lighting image data. The color conversion processing unit 1004 performs various processing such as white balance processing and gamma conversion processing on the lighting image data to generate final output image data.

  FIG. 11 shows a flowchart of a process of generating output image data by the lighting processing unit 306 in the present embodiment.

  First, in step S1101, the normal line calculation unit 1001 calculates normal line data representing the orientation of the surface with respect to all points in the scene based on the three-dimensional shape data. The normal line data can be calculated based on the direction of the surface calculated from the three-dimensional coordinates of each vertex constituting the surface of the polygon which is three-dimensional shape data. The method of calculating the normal line data is not limited, and other methods may be used.

  Next, in step S1102, the shadow image generation unit 1002 generates image data using computer graphics (CG) techniques for shadows that occur in a scene when virtual illumination is present at the illumination position set by the user. Calculated as Image data representing this shadow is referred to as shadow image data. The shadow image generation unit 1002 generates two types of image data of shadow image data and shade image data prior to generation of shadow image data.

  FIG. 12A shows an example of a shadow image 1210 generated in the same scene as FIG. In the shadow image 1210, a shadow 1211 is generated in an area other than the subject (that is, a background area) 602 by blocking the light from the illumination by the subject 601 in FIG. Shadowing is a phenomenon in which a point of interest is darkened because the light from the illumination is blocked by an object between the path of the point of interest and the light. Shadow image data is generated from the three-dimensional shape data of the scene and the illumination position. Various existing CG techniques can be applied to the generation of shadow image data, and for example, a technique called shadow mapping can be used.

  FIG. 12B shows an example of a shade image 1220 generated in the same scene as FIG. In the shade image 1220, light and shade (that is, shade) according to the orientation of the surface is generated in the area of the subject 601 and the area 602 other than the subject in FIG. A shade is the light and shade that occurs depending on the angle of incidence of light from the illumination to the point of interest when the light from the illumination falls on the point of interest. Shade image data is generated from the scene normal data and the illumination position. Various existing CG techniques can be applied to the generation of shade image data, and for example, a reflection model of Phong can be used.

Then, using the shadow image data I _shadow (i, j) and the shade image data I _shade (i, j), the shadow image generation unit 1002 generates shadow image data I _S (i, j) according to the following equation (12). Calculate). RGB values are stored as pixel values in each pixel of the shadow image data I _S , the shadow image data I _shadow and the shade image data I _shade .

  FIG. 12C shows an example of a shadow image 1230 generated from the shadow image 1210 of FIG. 12A and the shade image 1220 of FIG. 12B.

Next, in step S1103, the image combining unit 1003 combines lighting image data from color image data and shadow image data. The lighting image data I _L (i, j) is calculated based on the following equation (13) using the color image data I (i, j) and the shading image data I _s (i, j).

  FIG. 13 shows an example of a lighting image 1301 generated from the color image shown in FIG. 6C and the shaded image 1230 shown in FIG. 12C. According to this embodiment, as shown in FIG. 13, it is possible to add a shadow effect by virtual illumination to a color image.

  Next, in step S1104, the color conversion processing unit 1004 performs various color conversion processing on the lighting image data to generate final output image data. In this step, processing such as white balance adjustment, edge enhancement, and gamma correction is performed as color conversion processing. The image data subjected to the various color conversion processing is output to the image output unit 307 as output image data.

  As described above, the lighting processing by the lighting processing unit 306 is performed, and output image data in which a shading effect by virtual illumination is added to color image data is generated.

<Effect of this embodiment>
According to the present embodiment, when a virtual lighting effect is added to image data after shooting, it is possible to set the movable range of the illumination position based on the range of the object shape. That is, it is possible to set the position of the virtual illumination in consideration of the defect existing in the three-dimensional shape data of the subject.

  First, the problem caused by the loss of the subject region when the lighting effect, in particular the shadow effect, is given after the shooting by the lighting process will be described. The shadow is caused by the fact that the light emitted from the illumination is blocked by the object as described above. If the three-dimensional shape of the subject can be acquired correctly (that is, if there is no missing area in the three-dimensional shape data of the subject), the light emitted from the virtual illumination is correctly intercepted where the shadow occurs It is possible. However, if there is a missing area in the three-dimensional shape of the subject, the shadow can not be calculated correctly, and the shadow that would otherwise occur can not be reproduced. FIG. 14 shows the difference in the reproducibility of shadows depending on the presence or absence of a three-dimensional defect area of a subject. FIG. 14A illustrates the imaging device 101, the virtual illumination 1401, the subject 1402, and the background area 1403 as viewed from above. In FIG. 14A, the three-dimensional shape of the subject 1402 can be correctly acquired, and there is no missing area. In this case, the light emitted from the virtual illumination 1401 is blocked by the subject 1402 to generate a shadow in the area 1404 on the background area 1403. FIG. 14B shows an example of the lighting image generated in this case. As illustrated, it is possible to generate a lighting image in which the shadow is correctly reproduced by using a three-dimensional shape in which there is no missing area. Subsequently, with reference to FIGS. 14 (c) and 14 (d), an example where there is a missing area in the three-dimensional shape of the subject will be described. In FIG. 14C, the three-dimensional shape 1405 of the subject includes a defect area. In this case, the light emitted from the virtual illumination 1401 reaches the background area 1403 without being blocked in the missing area. As a result, on the background area 1403, a shadow is generated only in an area 1406 in which light is blocked by an area in which the three-dimensional shape 1405 of the subject can be acquired. FIG. 14D shows an example of the lighting image generated in this case. In this case, as shown in the figure, since a three-dimensional shape 1405 has a missing area, it is not possible to correctly reproduce the originally supposed shadow, and a lighting image including an unnatural shadow is generated. Such unnatural shadows are caused by the virtual illumination illuminating the three-dimensional defect area. On the other hand, even if there is a defect area in the three-dimensional shape of the subject, the virtual illumination only illuminates the area where the three-dimensional shape can be acquired, and if the defect area is not irradiated, unnatural shadows do not occur. . Therefore, in the present embodiment, it is possible to make an unnatural shadow less likely to occur by setting the irradiation position movable range of the virtual illumination so as to irradiate only the area where the three-dimensional shape of the subject can be correctly acquired. It is. Then, when the user sets the position of the virtual illumination, it is possible to prevent the setting of the position of the virtual illumination at the position where the unnatural shadow occurs by presenting the illumination position movable range.

Second Embodiment
The first embodiment has described the case where the movable range of the illumination position of the virtual illumination is set based on the range of the subject shape among the three-dimensional shape data used for the lighting process. However, in the first embodiment, since the background area in which the shadow is generated by the subject area is not considered, the generation of the unnatural shadow can not be accurately predicted. That is, the irradiation position movable range of the virtual illumination could not be set accurately. Thus, in the present embodiment, the movable range of the illumination position of the virtual illumination is set based on the positions of the imaging device, the subject, and the background area in addition to the range of the subject shape. In the present embodiment, since the generation position of the shadow can be predicted more accurately and the movable range of the illumination position can be set more accurately, generation of the unnatural shadow can be further suppressed.

  In the following, descriptions of configurations and operations similar to those of the first embodiment will be omitted, and configurations and operations different from the first embodiment will be mainly described.

<Appearance and Internal Configuration of Image Processing Device>
The appearance and internal configuration of the imaging apparatus 101 (that is, the image processing apparatus) in the present embodiment are the same as the appearance and internal configuration of the imaging apparatus 101 of the first embodiment described above with reference to FIGS. Therefore, the description is omitted.

<Configuration of Image Processing Unit 209>
FIG. 15 is a functional block diagram showing a configuration of the image processing unit 209 in the present embodiment. In the following, among the functional blocks shown in FIG. 15, a subject region extraction unit 1501 and an illumination position movable range setting unit 1502 which are different in operation from the first embodiment will be mainly described. The other functional blocks perform the same operations as in the first embodiment, and thus the description thereof is omitted.

  The subject region extraction unit 1501 in this embodiment separates color image data and three-dimensional shape data of a scene into a subject region and a background region, and extracts three-dimensional shape data of each of the subject region and the background region. Further, the illumination position movable range setting unit 1502 according to the present embodiment determines virtual illumination based on three-dimensional shape data of each of the subject area and the background area when it is determined that there is a defect in the three-dimensional shape data of the subject. Set the movable range for the lighting position of.

<Processing Flow of Image Processing Unit 209>
Subsequently, the processing flow of the lighting processing by each unit of the image processing unit 209 in the present embodiment will be described. FIG. 16 shows a flowchart of the lighting process in the present embodiment. In the processing flow, the operation is different from that of the first embodiment in two processes of extraction processing of a subject region in step S1601 and update processing of the illumination position movable range in step S1602. The other processes are the same as in the first embodiment, and therefore the description thereof is omitted, and the above two processes will be described below.

  In step S 1601, the subject region extraction unit 1501 separates the three-dimensional shape data of the scene into the subject region and the background region from the color image data and the three-dimensional shape data of the scene. Extract shape data. The same processing as that of the first embodiment can be applied to a method of extracting a subject region from three-dimensional shape data of a scene.

  In step S1602, the illumination position movable range setting unit 1502 sets the illumination position movable range based on the three-dimensional shape data of each of the subject region and the background region when there is a missing region in the three-dimensional shape data of the subject region. Make an update. The present embodiment differs from the update processing of the illumination position movable range of the first embodiment in that three-dimensional shape data of a background area is used in addition to three-dimensional shape data of a subject area.

  Hereinafter, the process of updating the illumination position movable range in the present embodiment will be described in detail. FIG. 17 is a schematic view showing the image pickup apparatus 101 and the scene as viewed from above. In FIG. 17A, the scene includes a subject area 1701 and a background area 1702. Of the three-dimensional shape of the subject area 1701, points 1703 and 1704 are three-dimensional points corresponding to the boundary between the subject area and the background area in the subject area. A method similar to that of Embodiment 1 can be used to calculate the three-dimensional point of this boundary. A straight line 1705 is a straight line passing through the imaging device 101 and the point 1703. A point 1706 is a three-dimensional point corresponding to an intersection of the background area 1702 and the straight line 1705. Similarly, a straight line 1707 is a straight line passing the imaging device 101 and the point 1704, and an intersection point of the straight line 1707 and the background region 1702 is a point 1708. Further, a straight line 1709 is a straight line connecting the point 1703 and the point 1708, and a straight line 1710 is a straight line connecting the point 1704 and the point 1706. If the three-dimensional position of the imaging apparatus 101 and the three-dimensional positions of the point 1703, the point 1704, the point 1706, and the point 1708 are given, a straight line equation representing the straight line 1709 and the straight line 1710 can be calculated.

  In this embodiment, an area surrounded by the subject area 1701, the straight line 1709, and the straight line 1710 is set as the movable range of the illumination position of the virtual illumination. However, in the present embodiment, the three-dimensional coordinates of the virtual illumination are represented by four variables of the rotation center position C, radius r, azimuth angle θ, elevation angle φ as described above, and among these, two variables of azimuth angle θ and elevation angle φ The movable range is set for. Therefore, the coordinates in the area surrounded by the subject area 1701, the straight line 1709, and the straight line 1710 are represented by four variables of the rotation center position C representing the illumination position described above, the radius r, the azimuth angle θ, and the elevation angle φ.

  Subsequently, a method of setting the movable range of the illumination position of the virtual illumination will be described with reference to FIG. A point 1711 represents the rotation center position C of the illumination position. In the present embodiment, the center of gravity OC of the subject region 1701 is set as the rotation center position C as described above. The arc 1712 is an arc of radius r centered on the point 1711. The value of the radius r is set by the illumination position setting unit 304 described above. A point 1713 represents an intersection of a straight line 1709 and the arc 1712, and a point 1714 represents an intersection of the straight line 1710 and the arc 1712. A point 1716 is an intersection point of a straight line 1715 connecting the point 1711 and the imaging device 101 with the arc 1712. The angle 1717 is an angle formed by three points 1713, 1711, and 1716. The angle 1718 is an angle formed by three points 1714, 1711, and 1716. If the three-dimensional position of the imaging apparatus 101 and the three-dimensional positions of the point 1703, the point 1704, the point 1706, and the point 1708 are given, it is possible to calculate the equation of the three-dimensional position of the point and the straight line, and the angle. It is possible.

Here, points 1713 to 1714 on the arc 1712 are movable range positions that can be taken by the illumination position of the virtual illumination in the case of the radius r described above. That is, the angle 1717 and the angle 1718 are movable ranges of the azimuth angle θ. Therefore, the angle theta ₁ of the formula (4) representing the movable range of the azimuth angle theta is next angle 1717, the angle theta ₂ is at an angle 1718.

In the above, the movable range of the azimuth angle θ has been described using FIG. 17 in which the subject and the imaging unit are viewed from the upper side. However, since the subject and virtual illumination are expressed in a three-dimensional space, the movable ranges φ ₁ and φ ₂ are calculated for the elevation angle φ as well as the azimuth angle θ.

  As described above, the process of updating the illumination position movable range in step S1602 is performed.

  As described above, in the present embodiment, the movable range of the illumination position is set by referring to not only the subject region of the three-dimensional shape data but also the background region. This makes it possible to more accurately predict the shadow generation position, thereby further suppressing the generation of unnatural shadows.

Third Embodiment
In the first and second embodiments, the movable range of the illumination position of the virtual illumination is set in order to suppress the generation of the unnatural shadow caused by the loss of the three-dimensional shape data of the subject. On the other hand, in the present embodiment, the setting range of the illumination size of the virtual illumination is to suppress generation of unnatural shadows that occur depending on the acquisition accuracy of the three-dimensional shape data of the object and the illumination size of the virtual illumination. Set For example, the lighting effect on the subject is different between a point light source with a small illumination size and a surface light source with a large illumination size, and an unnatural shadow may be generated depending on the illumination size. In addition, when the acquisition accuracy of the three-dimensional shape data is low, and there are irregularities that are not originally present on the surface of the subject, unnatural shadows that would otherwise not occur may be reproduced. Therefore, in the present embodiment, generation of an unnatural shadow is suppressed by setting the settable range of the illumination size of the virtual illumination in accordance with the acquisition accuracy of the three-dimensional shape data.

  In the following, descriptions of configurations and operations similar to those of the first embodiment will be omitted, and configurations and operations different from the first embodiment will be mainly described.

<Appearance and Internal Configuration of Image Processing Device>
The appearance and internal configuration of the imaging apparatus 101 (that is, the image processing apparatus) in the present embodiment are the same as the appearance and internal configuration of the imaging apparatus 101 of the first embodiment described above with reference to FIGS.

  In addition to the illumination position of the virtual illumination, the control unit 206 in the present embodiment controls the setting of the illumination condition including the illumination size of the virtual illumination. Specifically, when setting the illumination condition, the control unit 206 in the present embodiment determines an illumination condition update mode that indicates which parameter is to be updated among a plurality of parameters representing the illumination condition described later, and the illumination condition is determined. Control of the lighting condition determination flag indicating whether or not it is done. In the illumination condition update mode, 0 is set when the illumination size is updated, and 1 is set when the illumination position is updated. The illumination condition determination flag is set to 0 when the illumination condition is undecided and 1 when the illumination condition is decided. Both the illumination condition update mode and the illumination condition determination flag are given 0 as an initial value.

<Configuration of Image Processing Unit 209>
FIG. 18 is a functional block diagram showing a configuration of the image processing unit 209 in the present embodiment. The image processing unit 209 in the present embodiment receives raw image data from the A / D conversion unit 208, distance image data from the distance image acquisition unit 104, and illumination conditions of virtual illumination from the control unit 206. The image processing unit 209 performs lighting processing on the RAW image data based on the distance image data and the illumination condition. Then, the image processing unit 209 outputs an output image, which is the result of the writing process, to the PC / media 213 (from the encoder unit 210 via the media I / F 211) or to the display unit 105.

  In the following, among the functional blocks shown in FIG. 18, the functional blocks different from the first embodiment will be mainly described.

The illumination condition setting unit 1804 sets illumination conditions of virtual illumination including at least the illumination size according to the user operation input from the control unit 206. In the present embodiment, as the illumination condition, two pieces of information of the illumination size of the virtual illumination and the illumination position are set. FIG. 19 is a diagram for explaining the setting operation of illumination conditions and parameters in the present embodiment. As shown in FIG. 19A, a captured image 1901 to be subjected to lighting processing is displayed on the display screen of the display unit 105 of the imaging device 101, and an icon 1902 representing virtual illumination is superimposed on the captured image 1901. It is done. The user sets the illumination size and illumination position of the icon 1902 representing virtual illumination by a drag operation on the display unit 105. Further, as shown in FIG. 19B, in the present embodiment, the virtual illumination 1903 is represented by a sphere, and the size of the virtual illumination 1903 is represented by the radius Rs of the sphere. Further, the position of the virtual illumination 1903 is the rotation center position C of the virtual illumination 1903 C = (Cx, Cy, Cz), the rotation radius R _p of the virtual illumination 1903 around the rotation center position C, the rotation angle θ around the Y axis, This is expressed using four variables of the rotation angle φ around the Z axis. Hereinafter, the rotation angle θ about the Y axis is referred to as an azimuth angle, and the rotation angle φ about the Z axis is referred to as an elevation angle. With these four variables, the three-dimensional coordinates L = (Lx, Ly, Lz) of the illumination position of the virtual illumination can be determined using the following equation (14).

  The method of expressing the illumination size and the illumination position of the virtual illumination is not limited to the above, and other methods may be used.

  The imaging device condition acquisition unit 1805 acquires imaging conditions of the imaging device 101. The imaging conditions of the imaging apparatus 101 include the focal length of the lens, the number W of horizontal pixels of the color image data and the number H of vertical pixels of the color image data, and the position and orientation of the imaging apparatus 101 at the time of capturing color image data. Be The position and orientation of the imaging device 101 are calculated from color image data and three-dimensional shape data as described above in the first embodiment.

  The shape determination unit 1809 acquires the acquisition accuracy of the three-dimensional shape data of the subject region, and determines whether the accuracy is equal to or more than a predetermined accuracy. In the following, the case where the acquisition accuracy is equal to or more than a predetermined value is regarded as high accuracy, and the case where the acquisition accuracy is less than the predetermined value is regarded as low accuracy. Details of processing of this block will be described later.

The illumination size range setting unit 1810 can set the illumination size of the virtual illumination based on the acquisition accuracy of the three-dimensional shape data of the subject region when it is determined that the three-dimensional shape data of the subject region is low accuracy. Set Specifically, for the radius Rs, which is a parameter representing the illumination size, the lower limit value r _min is set as represented by the following equation (15).

In the present embodiment, the lower limit value is set as the illumination size range, but the upper limit value may be set together. The method of setting the lower limit value r _min will be described later.

  The illumination size range output unit 1811 outputs the setting range of the illumination size set by the illumination size range setting unit 1810 to the display unit 105 and presents the range to the user.

  The illumination size evaluation unit 1812 evaluates whether the user-specified illumination size set by the illumination condition setting unit 1804 is included in the settable range of the illumination size set by the illumination size range setting unit 1810.

  The warning output unit 313 in this embodiment is evaluated by the illumination size evaluation unit 1812 not to include the illumination size specified by the user within the settable range of the illumination size set by the illumination size range setting unit 1810. If it does, the warning is output to the display unit 105.

  As described above, the image processing unit 209 in the present embodiment has a function for performing lighting processing based on the lighting condition of virtual lighting including at least the lighting size.

<Processing Flow of Image Processing Unit 209>
Subsequently, the processing flow of the lighting processing by each unit of the image processing unit 209 in the present embodiment will be described. FIG. 20 shows a flowchart of the lighting process in the present embodiment. The processing of each step in the flowchart is implemented by the CPU 202 reading out and executing a program stored in the ROM 203 or the RAM 204. Note that, in the following, among the processing flows, processing of steps having operations different from those of Embodiment 1 will be mainly described.

  First, in step S2001, the image acquisition unit 301 acquires RAW image data, and in step S2002, the development unit 302 develops the RAW image data to generate color image data. Furthermore, in step S2003, the shape acquisition unit 303 acquires three-dimensional shape data (that is, polygons) of the imaging scene. The processes of steps S2001 to S2003 are the same as in the first embodiment.

  Next, in step S2004, the imaging apparatus condition acquisition unit 1805 acquires imaging conditions of the imaging apparatus 101. As described above, the imaging conditions of the imaging device 101 include the focal length of the lens, the number of pixels in each of the horizontal and vertical directions of color image data, and the position and orientation of the imaging device 101. The imaging device condition acquisition unit 1805 is a position of the imaging device 101, three-dimensional coordinates of an optical center of the imaging device, a posture of the imaging device 101 in the horizontal direction of the imaging device, a vertical direction of the imaging device, and an optical axis of the imaging device. Calculate each value of direction. In this embodiment, since the three-dimensional shape data is calculated from the distance value acquired by the distance image acquisition unit 104, the position of the imaging device 101 is the origin (0, 0, 0), and the horizontal direction of the imaging device is (1 , 0, 0), the vertical direction is (0, 1, 0), and the direction of the optical axis is (0, 0, 1).

  Next, in step S2005, the subject region extraction unit 308 extracts three-dimensional shape data of the subject region, and the process proceeds to step S2006.

In step S2006, the illumination size range setting unit 1810 initializes the settable range for the illumination size among the illumination conditions. Specifically, an initial value is given to the lower limit value r _min of the illumination size of the above-mentioned equation (15) as shown by the following equation (16).

  In step S2007, the shape determination unit 1809 obtains the acquisition accuracy of the three-dimensional shape data acquired by the shape acquisition unit 303, and determines whether the acquisition accuracy is less than a predetermined value. In the present embodiment, the acquisition accuracy of three-dimensional shape data is an index indicating how correctly the three-dimensional position of each vertex of the three-dimensional shape data can be acquired. The acquisition accuracy is calculated by comparing the correct three-dimensional shape data (that is, the original three-dimensional shape data) with the three-dimensional shape data acquired by the shape acquisition unit 303 with respect to the three-dimensional shape data whose shape is known. Ru. In the present embodiment, the direction of the normal is compared for each of the correct three-dimensional shape data and the acquired three-dimensional shape data, and the average error is taken as the acquisition accuracy.

  Hereinafter, a method of calculating acquisition accuracy in the present embodiment will be described with reference to FIG. FIG. 21 shows correct three-dimensional shape data 2101 whose shape is known, and three-dimensional shape data 2102 acquired by the shape acquiring unit 303. In the present embodiment, the above-described polygon is used as a constituent unit of the small areas 2103 and 2104 obtained by dividing each three-dimensional shape data. Each polygon of each three-dimensional shape data includes normals 2105 and 2106. The normal indicates a direction perpendicular to the surface and is expressed as a three-dimensional vector. The normals 2105 and 2106 represent directions by arrows starting from the center position of the polygon. The angle ψ (n) formed by the two vectors with respect to the normal Nr (n) of the correct three-dimensional shape data and the normal Na (n) of the acquired three-dimensional shape data in the nth polygon is And the error in the normal direction of the polygon.

  Then, an error in the normal direction is calculated for each polygon included in the subject region, and the average error ψ is obtained by the following equation (18).

  N represents the number of polygons included in the subject area. In this embodiment, the average error ψ in the normal direction of each polygon is used as an index indicating acquisition accuracy, and the acquisition accuracy is low when the average error 大 き い is large, and the acquisition accuracy is high when the average error 小 さ い is small. I assume.

  The predetermined value for determining the acquisition accuracy is acquired by storing in advance the value of the acquisition accuracy corresponding to the device or method used by the distance image acquisition unit 104 in the ROM 203 and reading it at the time of the processing of this step. May be The acquisition accuracy corresponding to the distance image acquisition unit 104 may be theoretically determined using a mathematical expression based on the design value of the distance image acquisition unit 104. The reference acquisition accuracy can also be obtained by measuring three-dimensional shape data of a subject whose three-dimensional shape is known by the distance image acquisition unit 104 at the time of manufacture or at any timing.

  As another method, there is also a method of acquiring acquisition accuracy by performing some processing on three-dimensional shape data used for lighting processing. However, in this case, when calculating the error in the normal direction of each polygon, since the correct normal direction is not known, the average normal direction in the polygons around the polygon of interest is treated as a pseudo normal. Calculate the acquisition accuracy.

  Then, the average error of the three-dimensional shape data acquired as described above is compared with a predetermined angle stored in advance in the ROM 203 to determine whether the acquisition accuracy is less than a predetermined value. That is, the acquisition accuracy in the case where the average error is equal to or more than the predetermined angle is determined to be low, while the acquisition accuracy in the case where the average error is less than the predetermined angle is determined to be high. When it is determined that the acquisition accuracy is low, the process proceeds to step S2008. On the other hand, when it is determined that the acquisition accuracy is high, the process proceeds to step S2010.

  In step S2008, when the acquisition accuracy of the three-dimensional shape data is low, the illumination size range setting unit 1810 updates the settable range of the illumination size from the three-dimensional shape data of the subject region.

In the present embodiment, the illumination size range setting unit 1810 calculates the lower limit value of the illumination size from the average error in the normal direction, and updates the settable range of the illumination size. Specifically, as shown in FIG. 22, the lower limit r _min is the illumination size at which the angle at which the virtual illumination is viewed from the rotation center position C of the virtual illumination is the average error 法線 in the normal direction determined in step S2007. Do. Therefore, the update value of the lower limit value r _min of the illumination size can be calculated using the following equation (19) from the average error ψ in the normal direction.

  Next, in step S2009, the illumination size range output unit 1811 outputs the setting range of the illumination size, and displays the range on the display unit 105. The display of the settable range of the illumination size is performed using figures and characters. For example, as shown in FIG. 23, a dotted line 2301 representing the lower limit size of illumination is displayed for an icon 1902 representing virtual illumination superimposed on the captured image 1901. The dotted line 2301 can be displayed as a concentric circle whose center position is the same as the icon 1902 representing virtual illumination at the current setting value. Further, the settable range of the illumination size may be displayed as characters 2302 in the display unit 105, for example.

  Next, in step S2010, the illumination condition setting unit 1804 sets initial values for the illumination condition of the virtual illumination, that is, the illumination size and illumination position of the virtual illumination. For the size of the virtual illumination, an initial value is set to the radius Rs of the virtual illumination. The initial value of the virtual illumination size may be a value within the settable range of the virtual illumination size set in step S2008 when the acquisition accuracy of the three-dimensional shape data is low. As for the position of the virtual illumination, initial values are set for each of four variables of the rotation center position C of the virtual illumination, radius r, azimuth angle θ, and elevation angle φ. As the rotation center position C of the virtual illumination, the center of gravity OC of the subject region of the three-dimensional shape data is set. The radius r, the azimuth angle θ, and the elevation angle φ can be set to any position within the movable range of the illumination position. For example, the position of the illumination that illuminates the subject from the front is set as the initial setting. For that purpose, the value of the Z component of the center of gravity OC of the subject area is set to the radius r, and 0 ° is set to each of the azimuth angle θ and the elevation angle φ. The Z component of the center of gravity OC of the subject area represents the distance from the imaging apparatus 101 to the subject.

  Next, in step S2011, the lighting processing unit 306 performs lighting processing to generate output image data. In the present embodiment, the lighting process is performed based on the illumination condition of the virtual illumination. Details will be described later.

  Next, in step S2012, the image output unit 307 outputs the generated output image data to the display unit 105 as in the first embodiment, and performs preview display of the lighting processing result. The user depresses the operation button 106 when judging that the desired shade can be reproduced by looking at the lighting processing result displayed as a preview. In the present embodiment, the lighting conditions are determined. At this time, the control unit 206 updates the lighting condition determination flag to one.

  Next, in step S2013, the image output unit 307 determines whether the illumination condition has been determined. Specifically, the image output unit 307 refers to the illumination condition determination flag, and proceeds to step S2014 if the value is 1 representing determination, and proceeds to step S2015 if 0 representing undetermined.

  In step S2014, the image output unit 307 outputs the output image data to the PC / media 213 for storage as in the first embodiment.

  In step S2015, the illumination condition setting unit 1804 sets two pieces of information of the size of the virtual illumination and the illumination position as the illumination condition. When the illumination condition update mode is 1, the illumination condition setting unit 1804 updates the radius Rs of the illumination indicating the size of the virtual illumination. Here, a new radius Rs is calculated using the amount of movement in the horizontal direction of the drag operation by the user. Specifically, Rs' is calculated from the pressing start coordinate Ps of the drag operation and the pressing end coordinate Pe using the following equation (20), and substituted into the radius Rs for update.

  W represents the number of pixels in the horizontal direction of color image data.

On the other hand, when the illumination condition update mode is 0, the illumination condition setting unit 1804 updates the azimuth angle θ and the elevation angle φ representing the position of the virtual illumination. Pressing the start coordinates on the image data _{Ps = (i s, j s} ), as the pressing end coordinate _{Pe = (i e, j e} ), calculates the theta 'and phi' using the following equation (21), The azimuth angle θ and the elevation angle φ are respectively substituted and updated.

  Here, A and B are expressed by the following equation (22), where u is the horizontal angle of view of the imaging device 101 and v is the vertical angle of view. As described above, W indicates the horizontal direction of color image data, and H indicates the number of pixels in the vertical direction of color image data.

Next, in step S2016, the illumination size evaluation unit 1812 determines whether the size of the virtual illumination specified by the user is included in the settable range of the illumination size described above. Here, it is determined whether the radius Rs of the illumination representing the illumination size is included in the settable range by comparing it with the lower limit value r _{min of the} radius set in step S2008. If the radius Rs is equal to or greater than the lower limit value r _min , the process proceeds to step S2011. On the other hand, when the radius Rs is less than the lower limit value r _min , the process moves to step S2017.

  In step S2017, the warning output unit 313 outputs a message (that is, character information) indicating that the illumination size specified by the user is less than the lower limit value or a warning including a diagram to the display unit 105 and displays it. FIG. 24 shows an example of a warning display in the present embodiment. FIG. 24 shows the case where the icon 1902 representing virtual illumination is inside the dotted line 2301 representing the lower limit size of illumination, and the illumination size smaller than the lower limit is set. In such a case, a warning message 2401 is displayed. In addition to displaying the warning message 2401, the color of the icon 1902 may be changed. When changing the color of the icon 1902, the color is switched between two types inside and outside the lower limit of the illumination size, and the color of the icon 1902 gradually approaches the border even within the illumination lower limit. You may change the The warning display method is not limited to these.

  As described above, the image processing unit 209 in the present embodiment performs the lighting process. By this processing, a lighting image in which a virtual lighting effect is added to the photographed color image data is generated.

<Operation of Lighting Processing Unit 306>
The operation of the lighting processing unit 306 in the present embodiment will be described in detail below. As described above, the lighting processing unit 306 performs lighting processing using color image data, three-dimensional shape data, imaging device position / posture data, and illumination conditions of virtual lighting, and further performs color conversion processing. To generate output image data. As described above, the illumination conditions include the size of the virtual illumination and the illumination position.

  FIG. 25 shows the configuration of the lighting processing unit 306 in the present embodiment. The lighting processing unit 306 in the present embodiment generates shadow image data using the illumination condition and the imaging device condition output from the illumination condition setting unit 1804 and the imaging device condition acquiring unit 1805 in the shading image generation unit 1002. , Different from the first embodiment. Details will be described below.

  FIG. 26 is a flowchart of a process of generating output image data by the lighting processing unit 306 in the present embodiment. The overall processing flow is the same as the processing flow described with reference to FIG. 11 in the first embodiment, but the processing by the shadow image generation unit 1002 in step S2602 is different among steps S2601 to S2604. Therefore, the process of step S2602 will be mainly described in detail below.

  First, in step S 2601, the normal vector calculation unit 1001 calculates normal vector data representing the orientation of the surface with respect to all points in the scene based on the three-dimensional shape data, and then the process proceeds to step S 2602.

  In step S2602, the shadow image generation unit 1002 generates, as image data, a shadow generated in a scene by virtual illumination (that is, virtual illumination) set by the user, using CG technology. In the present embodiment, image data representing this shadow (that is, shadow image data) is generated from three-dimensional shape data, imaging device conditions, and illumination conditions.

  There are generally two types of shades by CG techniques, as described above with reference to FIG. 12: shades and shadows. A shade is the light and shade that occurs depending on the angle of incidence of light from the illumination to the point of interest when the light from the illumination falls on the point of interest. On the other hand, the shadow is a phenomenon in which the point of interest is darkened by the light from the illumination being blocked by the object between the path of the point of interest and the illumination. In the present embodiment, at least a shade of shade may be generated.

First, the shade image in the present embodiment as shown in FIG. 12 (b) will be described. In the present embodiment, the shade image data is calculated from the normal data of the scene, the illumination position of the virtual illumination, and the illumination size of the virtual illumination. For example, when the reflection model of Phong is used and the virtual illumination is a point light source, the shade image I _shade (i, j) can be calculated by the following equation (23).

Here, L is a three-dimensional position of virtual illumination, P (i, j) is a position of three-dimensional shape data corresponding to coordinates (i, j), N (i, j) is a subject position P (i, j). The direction of the surface normal in j), α, represents the brightness of the virtual illumination. In this embodiment, the virtual illumination is not a point light source but has a size of radius Rs, so the following equation (24) obtained by modifying the above equation (23) to calculate the shade by light emitted from various places of illumination Calculate I _shade (i, j).

  According to the above equation (24), it is possible to calculate the shade of light incident on the subject position P (i, j) from the entire area of the virtual illumination having the size of the radius Rs.

  In the shade image data shown in FIG. 12B, light and shade corresponding to the direction of the surface occur in the subject area and the background area.

  Next, a shadow image in the present embodiment as shown in FIG. 12A will be described. In the present embodiment, the shadow image data is calculated from the three-dimensional shape data of the scene, the illumination size of the virtual illumination, and the illumination position of the virtual illumination. As described above, in the present embodiment, the shadow image data need not necessarily be generated. For example, a method called shadow mapping can be applied to generation of shadow image data. The shadow image data shown in FIG. 12A shows that shadows are generated in areas other than the subject as light is blocked by the subject.

Then, as in the first embodiment, the shadow image generation unit 1002 uses the equation (12) and uses the shade image data I _shade (i, j) and the shadow image data I _shadow (i, j). The shadow image data I _s (i, j) is calculated.

  Next, in step S2603, the image combining unit 1003 combines lighting image data from color image data and shadow image data as in the first embodiment. That is, as shown in FIG. 13 of the first embodiment, it is possible to add a shadow effect by virtual illumination to color image data.

  Next, in step S2604, the color conversion processing unit 1004 performs various color conversion processing on the above-described lighting image data as in the first embodiment, and generates final output image data. The image data subjected to the various color conversion processing is output to the image output unit 307 as output image data.

  As described above, lighting processing is performed by the lighting processing unit 306, and an output image is generated in which a shading effect by virtual lighting is added to color image data.

<Effect of this embodiment>
According to the present embodiment, when a virtual lighting effect is added to image data after shooting, it is possible to set the settable range of the illumination size based on the acquisition accuracy of the subject shape. That is, it is possible to set the size of the virtual illumination in accordance with the acquisition accuracy of the three-dimensional shape data of the subject.

  Conventionally, when performing lighting processing using low-accuracy three-dimensional shape data of an object, an unnatural shading effect occurs due to the unevenness included in the acquired three-dimensional shape data although it is not the original object. was there. The shading by the lighting process is calculated by adding up the shading by the light emitted from all the regions of the virtual illumination as described with reference to equation (24) in step S2602 of FIG. Therefore, the shadow effect changes depending on the size of the virtual illumination. For example, if the size of the virtual illumination is small, sharp and clear shadows will occur because less shadows are added. On the other hand, when the size of the virtual illumination is large, shadows due to light emitted from various positions are added, resulting in blurred shadows. Therefore, the shading effect due to the unevenness included in the three-dimensional shape data also becomes unnatural shading because it is clearly visible when the size of the virtual illumination is small. On the other hand, when the size of the virtual illumination is large, it is blurred and inconspicuous, so it is hard to be an unnatural shadow. Therefore, in the present embodiment, it is possible to make unnatural shadows less likely to occur by setting a lower limit on the size of the virtual illumination according to the acquisition accuracy of the three-dimensional shape data of the subject. Then, when the user sets the size of the virtual illumination, it is possible to prevent setting of the virtual illumination of the size causing the unnatural shadow by presenting the range of the size that can be set.

Fourth Embodiment
In the third embodiment, the case where the image pickup unit 102 and the distance image acquisition unit 104 are included in the housing of the image pickup apparatus 101 has been described, but in the present embodiment, the image pickup unit 102 and the distance image acquisition unit 104 are independent of each other The case where the configuration is the same will be described. In this embodiment, the color image data and the three-dimensional shape data may be acquired independently of each other, so the resolution of the lighting image may not correspond to the resolution of the polygon of the subject. Therefore, for example, when three-dimensional shape data is acquired at high resolution, a large number of polygons correspond to one pixel of the lighting image. In that case, if the error of the normal is calculated in polygon units of the three-dimensional shape data as in the third embodiment, the acquisition accuracy is more than necessary due to the error of the normal due to unevenness at a resolution finer than one pixel of the lighting image. It could have been underestimated. Therefore, in the present embodiment, the image capturing unit 102 and the distance image acquiring unit 104 are independent of each other, and the illumination size of the virtual illumination when the resolution of the three-dimensional shape data and the resolution of the color image data do not correspond. The method of setting the settable range will be described.

  In the following, descriptions of configurations and operations similar to those of the third embodiment will be omitted, and configurations and operations different from those of the third embodiment will be described.

<Configuration of Imaging System>
FIG. 27 shows a configuration of an imaging system in the present embodiment. In the imaging system in the present embodiment, a color image capturing unit (that is, a color image capturing apparatus) 2701 and a shape measuring unit (that is, a shape measuring apparatus) 2702 are configured independently of each other, and each image a subject 2703. The shape measurement unit 2702 measures three-dimensional shape data of the subject 2703 using a time-of-flight method, a multi-viewpoint camera method, a light cutting method, or the like. The three-dimensional shape data is a polygon composed of a point in a three-dimensional space and a surface connecting a plurality of three-dimensional points. The shape measuring unit 2702 may acquire three-dimensional shape data in a wide range by stitching three-dimensional shape data measured while moving around the subject 2703.

<Internal Configuration of Color Image Pickup Unit>
FIG. 28 is a block diagram showing an internal configuration of the color image pickup unit 2701 in the present embodiment. The color image pickup unit 2701 has a configuration similar to that of the image pickup apparatus 101 except that the distance image acquisition unit 104 exists outside as the shape measurement unit 2702 as compared with the image pickup apparatus 101 shown in FIG. The shape measuring unit 2702 transmits three-dimensional shape data to the color image capturing unit 2701 via a wired or wireless network. Alternatively, three-dimensional shape data may be passed via a removable storage medium. The image processing unit 209 in the present embodiment performs image processing using the three-dimensional shape data acquired by the shape measurement unit 2702. Details of the image processing will be described later.

<Configuration of Image Processing Unit 209>
FIG. 29 is a functional block diagram showing the configuration of the image processing unit 209 in the present embodiment. In the following, among the functional blocks shown in FIG. 29, the functional blocks different from the third embodiment will be mainly described.

  The shape acquisition unit 2903 in the present embodiment acquires three-dimensional shape data of the subject 2703 measured by the shape measurement unit 2702. The imaging apparatus condition acquisition unit 2905 acquires imaging conditions of the color image imaging unit 2701. The shooting conditions in the present embodiment include the focal length of the lens of the color image pickup unit 2701, the number of pixels in each of the horizontal and vertical directions of color image data, and the three-dimensional position / attitude of the color image pickup unit 2701 with respect to the subject 2703 It is. The shape determination unit 2909 calculates the acquisition accuracy of the three-dimensional shape data in consideration of the difference in resolution from the three-dimensional shape data of the subject region and the imaging device condition, and determines whether it is equal to or more than a predetermined acquisition accuracy.

<Processing Flow of Image Processing Unit 209>
Subsequently, the processing flow of the lighting processing by each unit of the image processing unit 209 in the present embodiment will be described. FIG. 30 shows a flowchart of the lighting process in the present embodiment. The processing of each step in the flowchart is implemented by the CPU 202 reading out and executing a program stored in the ROM 203 or the RAM 204. In the following, among the processing flows, the processing of the steps whose operation is different from that of the third embodiment will be mainly described.

  First, as in the third embodiment, in step S2001, the image acquisition unit 301 acquires RAW image data, and in step S2002, the development unit 302 develops the RAW image data to generate color image data.

  Next, in step S3003, the shape acquisition unit 2903 acquires three-dimensional shape data of a subject from the shape measurement unit 2702. As described above, here, the three-dimensional shape data of the subject is polygon data of the subject surface.

  Next, in step S3004, the imaging apparatus condition acquisition unit 2905 acquires the imaging conditions of the color image imaging unit 2701. As described above, the photographing conditions of the color image pickup unit 2701 which is an image pickup apparatus include the focal length of the lens, the number of pixels in each of the horizontal direction and the vertical direction of color image data, and the position / attitude of the color image pickup unit 2701 . The imaging apparatus condition acquisition unit 2905 acquires a rotation matrix R and a translation vector T of the camera as information representing the position and orientation of the color image pickup unit 2701. Here, the three-dimensional position / posture with respect to the subject 2703 can be obtained by a technique called SfM (Structure from Motion) which calculates the position / posture of the camera from a plurality of image data captured from different viewpoint positions. This is a technique for obtaining a projection transformation matrix by photographing with a camera from a plurality of corresponding points searched from color image data and three-dimensional shape data. Then, the rotation matrix R and translation vector T of the camera, that is, the position / orientation of the color image pickup unit 2701 can be calculated from the projective transformation matrix obtained. SfM is a well-known technology and is not the main point of the present invention, so the description is omitted.

  Next, as in the third embodiment, in step S2005, the subject area extraction unit 308 extracts three-dimensional shape data of the subject area, and in step S2006, the illumination size range setting unit 1810 sets the illumination size to one of the illumination conditions. Perform initialization of the settable range.

  Next, in step S3007, the shape determination unit 2909 calculates the acquisition accuracy of the three-dimensional shape data from the three-dimensional shape data and the imaging device condition, and determines whether the acquisition accuracy is less than a predetermined value. In the present embodiment, the color image data and the three-dimensional shape data may be acquired independently of each other, so that the resolution of the lighting image may not correspond to the resolution of the polygon of the subject. Therefore, for example, when three-dimensional shape data is acquired at a higher resolution, a large number of polygons correspond to one pixel of the lighting image. In that case, if the error of the normal is calculated in polygon units of shape data as in the third embodiment, the accuracy of acquisition is lower than necessary due to the error of the normal due to unevenness at a resolution finer than one pixel of the lighting image. It will be estimated. Therefore, the shape determination unit 2909 in the present embodiment extracts a region corresponding to each pixel of the lighting image from the three-dimensional shape data, and the angle between the normal of the region and the normal of the correct three-dimensional shape data forms 3 Calculate the acquisition accuracy of dimensional shape data.

For that purpose, first, the shape determination unit 2909 detects an area of three-dimensional shape data corresponding to each pixel of the lighting image. Writing image pixel (i, j) to detect an area of the three-dimensional shape data corresponding to the four straight lines l ₁ to l _4, represented by each of the following equation (25) to (28), The point of intersection with the three-dimensional shape data may be determined. Formulas (25) to (28) are the focal lengths f _x and f _y of the lens of the color image pickup unit 2701, the numbers W and H of pixels of the color image data in the horizontal and vertical directions, the position of the color image pickup unit 2701 It is expressed using a rotation matrix R representing a posture and a translation vector T.

Here, f _x and f _y are focal lengths expressed in pixel units of the color image pickup unit 2701. l _{1 to} l ₄ represent straight lines using the parameter s. The intersections of the straight lines l _{1 to} l ₄ and the three-dimensional shape data can be determined using collision determination between a polygon and a straight line used in ray tracing in the CG technique. In this case, among the polygons included in the three-dimensional shape data, a polygon having an intersection with the straight lines l _{1 to} l ₄ is searched. A plane formed by these four intersections is taken as an area corresponding to a pixel of color image data.

  Next, the shape determination unit 2909 calculates the normal to the surface of the area corresponding to the pixel of the color image data. In addition, since the intersection of four points does not necessarily exist on one plane, the normal line of the plane closest to the four points obtained by optimization may be calculated. Alternatively, a quadrilateral of four points may be divided into two diagonal triangles, and the average value of the normals to the plane of each triangle may be treated as the normal of the plane that the four points make. A corresponding area is detected from the three-dimensional shape data for all the pixels in the subject area of the color image data, and the normal is calculated.

  Next, the shape determination unit 2909 calculates an angle formed by the normal to the surface of the area corresponding to each pixel of the color image data and the normal to the same place in the correct three-dimensional shape data. The angle between the two normals can be calculated using the above equation (18) of the third embodiment. This angle is an error in the normal direction in the area corresponding to the pixel of the color image data. When the correct three-dimensional shape data is not included, the normal direction of the average of the peripheral area of the attention area is treated as a pseudo normal to calculate the acquisition accuracy. Then, the average value 誤差 of the errors in the normal direction calculated in the area corresponding to the pixel included in the subject area of the color image data is used as an index indicating acquisition accuracy. As in the third embodiment, when the average error is large, the acquisition accuracy is low, and when the average error is small, the acquisition accuracy is high.

  Then, the shape determination unit 2909 compares the acquisition accuracy of the three-dimensional shape data acquired as described above with a predetermined angle stored in advance in the ROM 203, and determines whether the acquisition accuracy is less than a predetermined value. Do. That is, when the average error is equal to or more than the predetermined angle, it is determined that the acquisition accuracy is low, and the process proceeds to step S2008. On the other hand, if the average error is less than the predetermined angle, it is determined that the acquisition accuracy is high, and the process proceeds to step S2010.

  The other steps are the same as in the third embodiment.

  As described above, in the writing process according to this embodiment, the processes in step S3003, step S3004, and step S3007 are different from those in the third embodiment.

  As described above, in the present embodiment, the method of setting the settable range of the size of the virtual illumination in the case where the image pickup unit 102 and the distance image acquisition unit 104 are independent has been described. In the present embodiment, even when the resolution of the three-dimensional shape data and the resolution of the color image data do not correspond to each other, it is possible to set an appropriate settable range for the size of the virtual illumination.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Processing is also feasible. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

101 image processing apparatus 105 display unit 209 image processing unit 301 image acquisition unit 302 development unit 303 shape acquisition unit 304 illumination position setting unit 305 imaging device position / attitude calculation unit 306 writing processing unit 307 image output unit 308 object area extraction unit 309 shape Judgment unit 310 Illumination position movable range setting unit 311 Illumination position movable range output unit 312 Illumination position evaluation unit 313 Warning output unit

Claims (23)

An image processing apparatus that adds a shadow effect by virtual illumination to the captured image based on three-dimensional shape data corresponding to the captured image,
Extracting means of the subject area for extracting three-dimensional shape data of the subject area from the three-dimensional shape data;
An image characterized by comprising movable range setting means for setting a movable range of the virtual illumination based on the loss of the three-dimensional shape data when there is a loss in the three-dimensional shape data of the subject region Processing unit.   The movable range setting means sets the movable range based on the center of gravity of the subject region in the three-dimensional shape data and the boundary between the subject region and the background region corresponding to the defect. An image processing apparatus according to Item 1.   The movable range setting means sets the movable range based on three-dimensional shape data of a background area and a three-dimensional position of an imaging device in addition to the three-dimensional shape data of the subject area. The image processing apparatus according to claim 1.   The image processing apparatus according to any one of claims 1 to 3, further comprising shape determination means for determining whether or not the defect exists in the three-dimensional shape data of the subject region.   5. The image processing apparatus according to claim 4, wherein the shape determination unit determines that the defect exists when the three-dimensional shape data of the subject region can not be acquired over the entire circumference of the subject.   The shape determination means determines whether or not the defect exists by determining whether a polygon forming the subject area is a closed surface or an open surface. The image processing apparatus according to 4.   The shape determination means may determine whether the defect exists by using three-dimensional shape data of the subject region and a result obtained by applying an image recognition technique to the captured image. The image processing apparatus according to claim 4.   The image processing apparatus according to any one of claims 1 to 7, further comprising display means for displaying a movable range of the illumination position of the virtual illumination.   The display means compares the illumination position of the virtual illumination with the movable range of the illumination position of the virtual illumination, and the illumination position of the virtual illumination is near the boundary of the movable range of the illumination position of the virtual illumination or 9. The image processing apparatus according to claim 8, wherein a warning is presented to the user when it is set outside the movable range.   The display means presents a warning to the user by displaying character information for the warning, or changing the color of an icon representing the illumination position of the virtual illumination. The image processing apparatus according to claim 9, wherein   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 10. An image processing method by an image processing apparatus, which adds a shadow effect by virtual illumination to a captured image based on three-dimensional shape data corresponding to the captured image,
Extracting three-dimensional shape data of a subject area from the three-dimensional shape data;
And setting a movable range of the virtual illumination based on the loss of the three-dimensional shape data, when there is a loss in the three-dimensional shape data of the subject region. An image processing apparatus that adds a shadow effect by virtual illumination to the captured image based on three-dimensional shape data corresponding to the captured image,
Extracting means of the subject area for extracting three-dimensional shape data of the subject area from the three-dimensional shape data;
Illumination size range setting means for setting the settable range of the illumination size of the virtual illumination based on the acquisition accuracy of the three-dimensional shape data when the acquisition accuracy of the three-dimensional shape data of the subject region is low An image processing apparatus characterized by   The acquisition accuracy of the three-dimensional shape data is calculated based on an angle formed by a normal line in the three-dimensional shape data of the subject region and a normal line in the original three-dimensional shape data. The image processing apparatus according to claim 1.   When the resolution of the three-dimensional shape data of the subject region is higher than the resolution of the captured image, the normal in the three-dimensional shape data of the subject region is the normal to the region corresponding to the pixel of the captured image. The image processing apparatus according to claim 13, characterized in that   The acquisition accuracy of the three-dimensional shape data is calculated based on an angle formed by the normal line of the attention area in the three-dimensional shape data of the subject area and the normal line having the average normal direction in the peripheral area of the attention area. The image processing apparatus according to claim 13, characterized in that:   The image processing apparatus according to any one of claims 13 to 16, further comprising shape determination means for determining whether the acquisition accuracy of the three-dimensional shape data is lower than a predetermined value.   The illumination size range setting means may set the illumination size setting range with a radius representing the illumination size as a lower limit when the angle from which the virtual illumination is viewed from the center of gravity of the subject area is an angle representing the acquisition accuracy. The image processing apparatus according to any one of claims 13 to 17, wherein   The image processing apparatus according to any one of claims 13 to 18, further comprising display means for displaying the settable range of the illumination size of the virtual illumination.   The display means compares the illumination size of the virtual illumination with the setting range of the illumination size of the virtual illumination, and the illumination size of the virtual illumination is near the boundary of the setting range of the illumination size of the virtual illumination or 20. The image processing apparatus according to claim 19, wherein a warning is presented to the user when being set outside the settable range.   The display means presents a warning to the user by displaying text information for the warning, or changing the color of an icon representing the size of illumination of the virtual illumination. 21. The image processing apparatus according to claim 20, wherein   A program for causing a computer to function as the image processing apparatus according to any one of claims 13 to 21. An image processing method by an image processing apparatus, which adds a shadow effect by virtual illumination to a captured image based on three-dimensional shape data corresponding to the captured image,
Extracting three-dimensional shape data of a subject area from the three-dimensional shape data;
And setting the setting range of the illumination size of the virtual illumination based on the acquisition accuracy of the three-dimensional shape data when the acquisition accuracy of the three-dimensional shape data of the subject region is low. Image processing method.