JP2008204318A - Image processor, image processing method and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of dynamically determining a reflection parameter close to an actual object as a reflection parameter of a surface of a subject model. <P>SOLUTION: A three-dimensional model creation section 22 creates a three-dimensional model of a subject from a plurality of subject images obtained by photographing the subject. A partial region selection section 23 divides the surface of the three-dimensional model into a plurality of surface regions and selects a partial photographed image of a corresponding region corresponding to each surface region for each subject image. A partial model image creation section 26 reads a reflection parameter of each partial region from a material database 25 and creates a partial model image corresponding to each partial photographed image using the read reflection parameter and illumination conditions under which the subject image was photographed. A parameter retrieval section 27 calculates the goodness of fit of partial model images for partial photographed images for each surface region and determines the reflection parameter of the partial region corresponding to the partial model image with the highest goodness of fit as the reflection parameter of each surface region. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for determining a reflection parameter of a surface of a subject model, and in particular, reflection on the surface of a three-dimensional subject model created from a two-dimensional image photographed by a multi-view camera. The present invention relates to an image processing apparatus, an image processing method, and an image processing program for determining parameters.

  Photographing a moving subject, such as a traditional dancer, in a fixed lighting environment with multiple cameras and using the visual volume intersection method, etc., the three-dimensional shape of the subject and the color and pattern of the shape surface (hereinafter referred to as the texture) (Refer to Non-Patent Document 1, for example) to obtain a moving three-dimensional CG (computer graphics) model.

The three-dimensional model created by the above processing is arranged with a separately prepared CG object such as a stage or another performer as a component to be arranged in a virtual scene in the CG environment, and is rendered as a CG image. At this time, the texture data mapped to the three-dimensional model has sufficient resolution and reflection parameters close to those of the real object in order to fuse the three-dimensional model of the subject with other objects in the CG space without any sense of incongruity. It is desirable.
Kashiyama et al., “Surface texture mapping method to 3D moving object depending on display viewpoint”, Information Processing Society of Japan, Computer Vision and Image Media Study Group, vol. 145, pp. 17-24, September 2004

  However, since the image data used for the visual volume intersection method is captured under a specific illumination environment and includes a reflection component peculiar to the environment, the reflection parameter can be dynamically separated from the captured image. It is difficult to dynamically obtain a reflection parameter closer to a real object.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of dynamically determining a reflection parameter close to a real object as a reflection parameter of the surface of a subject model.

  An image processing apparatus according to the present invention is an image processing apparatus that determines a reflection parameter of a surface of a subject model, and a database unit that stores a reflection parameter in advance for each partial region on the surface of a subject, and a plurality of different mutually An acquisition unit that acquires a plurality of subject images obtained by photographing the subject from a camera viewpoint, a model creation unit that creates a three-dimensional model of the subject from the plurality of subject images, and a surface of the three-dimensional model as a plurality of surface regions A selection unit that divides and selects a partial captured image of a corresponding region corresponding to each surface region for each of the subject images, reads reflection parameters of each partial region from the database unit, reads the reflected parameters and the plurality of subject images An image creation unit that creates a partial model image corresponding to each partial captured image using the lighting conditions in which A determination unit that calculates a degree of matching of the partial model image with respect to the partial captured image for each surface area, and determines a reflection parameter of the partial area corresponding to the partial model image having the highest degree of matching as a reflection parameter of each surface area; Is provided.

  In the image processing apparatus according to the present invention, a plurality of subject images obtained by photographing subjects from a plurality of different camera viewpoints are acquired, a three-dimensional model of the subject is created from the plurality of subject images, and the surface of the three-dimensional model is A partial photographed image of a corresponding area corresponding to each surface area is selected for each subject image. Here, in the database unit, the reflection parameter is stored in advance for each partial area obtained by dividing the surface of the subject into a plurality of parts, and the reflection parameter of each partial area is read out from the database unit. A partial model image corresponding to each partial captured image is created using the illumination conditions under which the subject image was captured. Thereafter, the suitability of the partial model image with respect to the partially captured image is calculated for each surface area, and the reflection parameter of the partial area corresponding to the partial model image having the highest fitness is determined as the reflection parameter of each surface area. A reflection parameter close to a real object can be dynamically determined as the reflection parameter of the model surface.

  The database unit stores in advance reflection parameters calculated from an image of the subject photographed at a first resolution, and the acquisition unit captures a plurality of photographs of the subject at a second resolution lower than the first resolution. Preferably, a subject image is acquired, and the image creation unit creates a low-resolution partial model image obtained by converting the resolution from the first resolution to the second resolution as the partial model image.

  In this case, even when a low-resolution subject image is acquired and a high-resolution reflection parameter is stored in the database unit, the suitability of the partial model image with respect to the partial captured image is calculated with high accuracy in accordance with the resolution of the subject image. Therefore, the reflection parameter closer to the real object can be dynamically determined as the reflection parameter of the surface of the subject model.

  The determining unit calculates a degree of matching of the low-resolution partial model image with respect to the partial captured image, determines a predetermined number of partial regions having a high degree of matching as candidate partial regions, and the image creating unit is configured to select the candidate partial region. A subpixel partial model image having a third resolution higher than the second resolution from the reflection parameter of the second resolution, and the determining unit calculates a degree of fitness of the subpixel partial model image with respect to the partial captured image, Preferably, the reflection parameter of the partial area corresponding to the subpixel partial model image having the highest is determined as the reflection parameter of each surface area.

  In this case, after calculating the suitability of the low-resolution partial model image with respect to the partial captured image, a subpixel partial model image is created from a predetermined number of candidate partial regions having a high fitness, and the subpixel partial model image is compared with the partial captured image. Since the degree of matching is calculated and the reflection parameter of the partial area corresponding to the subpixel partial model image with the highest degree of matching is determined as the reflection parameter of each surface area, after narrowing down the candidate partial areas, The matching degree can be obtained, and the reflection parameter closer to the real object can be determined in a short time as the reflection parameter of the surface of the subject model.

  The database unit preferably stores in advance a diffuse reflection parameter and a specular reflection parameter as the reflection parameter.

  In this case, since the diffuse reflection parameter and the specular reflection parameter can be obtained as the reflection parameters of the surface of the subject model, a more realistic texture can be dynamically generated using these parameters.

  The database unit stores in advance the specular reflection parameter for each partial region with a constant specular reflection parameter in the partial region, and the image creation unit supports each partial captured image with the specular reflection parameter in the partial region constant. It is preferable to create a partial model image to be created.

  In this case, the storage capacity of the database unit can be reduced, and a partial model image can be created at high speed.

  An image processing method according to the present invention includes a database unit that stores reflection parameters in advance for each partial region obtained by dividing the surface of a subject into a plurality of parts, an acquisition unit, a model creation unit, a selection unit, an image creation unit, An image processing method for determining a reflection parameter of a surface of a subject model using an image processing device including a determination unit, wherein the acquisition unit captures the subject from a plurality of different camera viewpoints. Acquiring the image, the model creating unit creating a three-dimensional model of the subject from the plurality of subject images, and the selecting unit dividing the surface of the three-dimensional model into a plurality of surface regions, Selecting a partial captured image of a corresponding region corresponding to each surface region for each of the subject images, and the image creating unit reflecting each partial region from the database unit A step of creating a partial model image corresponding to each partial captured image using the read reflection parameter and the illumination condition in which the plurality of subject images are captured; and the determination unit includes: And calculating the degree of matching of the partial model image with respect to the partial captured image, and determining the reflection parameter of the partial area corresponding to the partial model image having the highest degree of matching as the reflection parameter of each surface area.

  An image processing program according to the present invention is an image processing program for determining a reflection parameter of a surface of a subject model, and a database unit that stores a reflection parameter in advance for each partial region obtained by dividing the surface of a subject. An acquisition unit that acquires a plurality of subject images obtained by photographing the subject from a plurality of different camera viewpoints, a model creation unit that creates a three-dimensional model of the subject from the plurality of subject images, and a surface of the three-dimensional model Is divided into a plurality of surface regions, a selection unit that selects a partial captured image of a corresponding region corresponding to each surface region for each of the subject images, and reflection parameters of each partial region are read from the database unit, and the read reflection parameters And a partial model image corresponding to each partial captured image using the lighting conditions in which the plurality of subject images are captured. And an image creation unit that creates a matching degree of the partial model image with respect to the partial captured image for each surface area, and sets a reflection parameter of the partial area corresponding to the partial model image with the highest matching degree for each surface area. The computer is made to function as a determination unit that determines the reflection parameter.

  According to the present invention, for each subject image, a partial captured image of a corresponding region corresponding to each surface region is selected, and the reflection parameter read from the database unit and the illumination conditions under which the plurality of subject images are captured are used. A partial model image corresponding to each partial captured image is created, and the degree of suitability of the partial model image with respect to the partial captured image is calculated for each surface region, and the reflection parameter of the partial region corresponding to the partial model image having the highest fitness is determined. Since it is determined as the reflection parameter of each surface region, the reflection parameter close to the real object can be dynamically determined as the reflection parameter of the surface of the subject model.

  An image processing apparatus according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention.

  1 includes a plurality of video cameras 11 to 1K, an image acquisition unit 21, a three-dimensional model creation unit 22, a partial region selection unit 23, an input unit 24, a material database 25, a partial model image creation unit 26, A parameter search unit 27, a texture update unit 28, and a CG creation unit 29 are provided.

  The K video cameras 11 to 1K (K is an integer of 2 or more) are arranged at predetermined intervals on a circle centered on the subject, and photograph a moving subject, for example, a person from a plurality of different camera viewpoints. The subject image is output to the image acquisition unit 21. For example, ten video cameras are used as the video cameras 11 to 1K, and color moving images photographed with 300,000 pixels are output from each video camera to the image acquisition unit 21.

  The image acquisition unit 21, the 3D model creation unit 22, the partial region selection unit 23, the input unit 24, the material database 25, the partial model image creation unit 26, the parameter search unit 27, the texture update unit 28, and the CG creation unit 29 An input device, a display device, a ROM (Read Only Memory), a CPU (Central Processing Unit), a RAM (Random Access Memory), an image I / F (Interface) unit, an external storage device, etc. This is realized by executing an image processing program for performing image processing, which will be described later, stored in advance in an external storage device by the CPU.

  Note that the image acquisition unit 21, the three-dimensional model creation unit 22, the partial region selection unit 23, the input unit 24, the material database 25, the partial model image creation unit 26, the parameter search unit 27, the texture update unit 28, and the CG creation unit 29. The configuration example is not particularly limited to the above example, and each block is configured by dedicated hardware, or only a part of blocks or only a part of processing in the block is configured by dedicated hardware. Various modifications are possible. Alternatively, the material database may be connected to a computer via a predetermined network, and only the image processing described later may be executed by the computer.

  The image acquisition unit 21 acquires a plurality (K) of subject images obtained by capturing subjects from a plurality of different camera viewpoints, temporarily stores the acquired plurality of subject images, and three-dimensionally stores the plurality of subject images. Output to the model creation unit 22. The three-dimensional model creation unit 22 creates a three-dimensional model of a subject from a plurality of subject images using, for example, a view volume intersection method. Note that the method of creating the three-dimensional model is not particularly limited to the above example, and various modifications such as the combined use of the visual volume intersection method and the stereo matching method are possible.

  The partial region selection unit 23 reads the three-dimensional model from the three-dimensional model creation unit 22 and divides the surface of the three-dimensional model into a plurality of surface regions. For example, when the three-dimensional model is composed of a plurality of polygons, one polygon can be divided as one surface area, or one polygon can be divided into a plurality of surface areas. The partial region selection unit 23 selects one surface region i (i = 1,..., N) from among a plurality of divided (N: N is an integer of 2 or more) surface regions, A three-dimensional position and orientation are calculated, and a two-dimensional position on each subject image corresponding to the surface area i is calculated from the three-dimensional position and orientation. The partial area selection unit 23 refers to the image acquisition unit 21 that temporarily stores a plurality of subject images, and for each subject image, the partial captured image j (j = 1, 1) of the corresponding region corresponding to the surface region i. ..., K) are selected and output to the parameter search unit 27.

  The input unit 5 is used for a user to input various data. For example, photographing illumination conditions when a subject is photographed by the video cameras 11 to 1K are input via the input unit 5, and this photographing illumination is used. The condition is input to the partial model image creation unit 26. The input unit 5 is also used to instruct the texture update unit 28 about a texture update method.

  The material database 25 stores a reflection parameter in advance for each partial area obtained by dividing the surface of the subject into a plurality of parts. Specifically, an object (subject) that may be present in the scene at the time of shooting is shot with a plurality of cameras in an environment where the light source position is known, and a plurality of high resolution (for example, 6 million pixels) shot The specular reflection parameter and the high-resolution diffuse reflection parameter (color information) are obtained for each partial region from the still image, and the obtained data is stored in advance in the material database 25 as material data. The size and shape of the partial area can be arbitrarily determined according to the ability of the computer functioning as the image processing apparatus.

Here, as the material data, the diffuse reflection parameters [k _{d, r (x, y)} , k _{d, g in} the coordinate set (partial region) R, (x, y) ∈R of the diffuse reflection parameters included in the data are included. _{(X, y)} , k _{d, b (x, y)} ], and the specular reflection parameter k _s in the minute region R is stored in a database for each partial region. Here, the diffuse reflection parameter is stored for each position of the R, G, and B color components in the minute region R, and the specular reflection parameter is stored in the minute region R in order to reduce the storage capacity and simplify the processing. Then, it is assumed that it is constant.

  In order to obtain an optimal solution that takes into account the positional deviation between the partial model image and the partial captured image in the reflection parameter search process described later, a large amount of material data in which the coordinate set R is translated must be generated. Actually, the diffuse reflection parameter is shared by a plurality of material data, and only the movement amount of the coordinate set R is separately recorded, so that the storage area can be used efficiently.

  The partial model image creation unit 26 reads the reflection parameters of each partial region from the material database 25, and calculates the read reflection parameters, the illumination conditions of the subject image input via the input unit 24, and the partial region selection unit 23. Using the three-dimensional position and orientation of the surface area i thus created, partial model images 1 to K corresponding to the partial captured images 1 to K are created for each surface area i. At this time, the partial model image creation unit 26, for example, converts the resolution of the partial model image from a resolution of 6 million pixels to a resolution of 300,000 pixels so that the resolution of the partial model image matches the resolution of the partial captured image. The resolution partial model images 1 to K are created.

  The parameter search unit 27 uses the partial captured images 1 to K selected by the partial region selection unit 23 and the low resolution partial model images 1 to K created by the partial model image creation unit 26 for each surface region. The degree of matching of the partial model image with respect to the partial captured image is calculated, and the reflection parameter of the partial area corresponding to the partial model image with the highest degree of matching is determined as the reflection parameter of each surface area.

  Specifically, the parameter search unit 27 calculates the degree of matching of the low-resolution partial model image with respect to the partial captured image as described above, and the part associated with the reflection parameter that is the basis of the low-resolution partial model image Among the regions, the top M partial regions (M is a predetermined integer) having the highest fitness are determined as candidate partial regions.

  Thereafter, the parameter search unit 27 instructs the partial model image creation unit 26 to create subpixel partial model images for the top M candidate partial regions, and the partial model image creation unit 26 selects the top M candidate partial regions. A subpixel partial model image having a resolution higher than the resolution of the partial captured image is created from the reflection parameter, and the subpixel partial model image is output to the parameter search unit 27. The parameter search unit 27 calculates the degree of matching of the subpixel partial model image with respect to the partial captured image, and finally uses the reflection parameter of the partial area corresponding to the subpixel partial model image with the highest degree of matching as the reflection parameter of each surface area. To decide.

  In this way, in order to obtain a correspondence between the partial captured image and the partial model image (material data) obtained at the time of shooting, first, the material data in the material database 25 is a low resolution degraded image (low resolution partial model) according to the shooting conditions. Next, the partial captured image that is the target area of the subject image is compared with the low-resolution partial model image, and the position parameter is further set for the top M model images that have given high score fitness. Can be estimated with sub-pixel accuracy, and finally, a partial model image having the highest fitness, that is, material data can be selected.

  The texture update unit 28 reads the 3D model from the 3D model creation unit 22 and uses the reflection parameters of the respective surface regions determined by the parameter search unit 27 according to the texture update method instructed via the input unit 5. To dynamically update the texture of the 3D model. For example, the texture update unit 28 updates the reflection parameter and reconstructs the texture image using the Back Projection method.

  The CG creation unit 29 creates a CG image using a three-dimensional model with an updated texture as a component to be placed in a virtual scene in the CG environment. For example, when the subject is a Noh performer, a 3D model of the Noh performer is arranged together with a separately prepared CG object such as a stage or another performer, and rendered as a CG image. Note that the texture update unit 28 may not update the texture image when the degree of matching is less than a predetermined threshold. In this case, rendering using only the information of the captured image is performed.

  In this way, in the present embodiment, the reflection characteristics of the object in the scene are modeled in advance and stored in the database, and the reflection parameter matching the surface texture of the three-dimensional model obtained at the time of shooting is searched in the database. The reflection parameter of the three-dimensional model can be updated dynamically.

  In the present embodiment, the material database 25 corresponds to an example of a database unit, the image acquisition unit 21 corresponds to an example of an acquisition unit, the 3D model creation unit 22 corresponds to an example of a model creation unit, and a partial region selection The unit 23 corresponds to an example of a selection unit, the partial model image generation unit 26 corresponds to an example of an image generation unit, and the parameter search unit 27 corresponds to an example of a determination unit.

  Next, the reflection parameter update process for updating the reflection parameter on the surface of the subject model by the image processing apparatus configured as described above will be described. FIG. 2 is a flowchart for explaining the reflection parameter update process by the image processing apparatus shown in FIG. 1, FIG. 3 is a diagram schematically showing the reflection parameter search process shown in FIG. 2, and FIG. 5 is a diagram illustrating an example of material data, FIG. 5 is a diagram illustrating an example of a subject image, FIG. 6 is a diagram illustrating an example of a three-dimensional model, and FIG. 7 is an example of a specular reflection parameter. FIG. 8 is a diagram showing an example of a partially captured image, FIG. 9 is a diagram showing an example of a CG image with updated texture, and FIG. 10 shows details depending on whether or not the texture is updated. It is a figure which shows an example of a high-definition CG image.

  Here, in the material database 25, specular reflection parameters and diffuse reflection parameters are stored in advance as material data for each partial region obtained by dividing the surface of the subject into a plurality of parts. For example, a plurality of rectangular shapes shown in FIG. Are stored in the material database 25 in advance.

  Referring to FIG. 2, first, in step S <b> 11, each video camera 11 to 1 </ b> K outputs the captured subject image to the image acquisition unit 21, and the image acquisition unit 21 acquires a plurality of subject images and performs three-dimensional processing. Output to the model creation unit 22. For example, a plurality of subject images shown in FIG. 5 are output to the three-dimensional model creation unit 22.

  Next, in step S <b> 12, the three-dimensional model creation unit 22 creates a three-dimensional model of the subject from the plurality of subject images using the view volume intersection method, and outputs the created three-dimensional model to the partial region selection unit 23. To do. For example, a three-dimensional model shown in FIG. 6 is created from a plurality of subject images shown in FIG.

  Next, in step S13, the partial region selection unit 23 reads the three-dimensional model from the three-dimensional model creation unit 22, divides the surface of the three-dimensional model into a plurality of surface regions, and among the plurality of divided surface regions. To select one surface area i (i = 1,..., N).

  Next, in step S14, the partial region selection unit 23 calculates the three-dimensional position and orientation of the surface region i (for example, the three-dimensional coordinates of the center position of the surface region i and the normal direction angle of the surface region i). To the partial model image creation unit 26. Next, in step S15, the partial region selection unit 23 calculates a two-dimensional position on each subject image corresponding to the surface region i from the calculated three-dimensional position and orientation.

  Next, in step S <b> 16, the partial region selection unit 23 determines a corresponding region corresponding to the surface region i from the two-dimensional position on the subject image and the size of the surface region i for each subject image, and the image acquisition unit 21. Referring to each subject image, a partial captured image j (j = 1,..., K) in a corresponding region corresponding to the surface region i is extracted and output to the parameter search unit 27.

  Next, in step S17, the partial model image creation unit 26 and the parameter search unit 27 search the reflection parameter most similar to the surface region i from the material database 25 as follows.

  First, the partial model image creation unit 26 reads the reflection parameters of each partial region from the material database 25, the read reflection parameters, the illumination conditions of the subject image input via the input unit 24, and the partial region selection unit 23. Using the three-dimensional position and orientation of the surface region i calculated by the above, partial model images 1 to K corresponding to the partial captured images are created.

  Next, the parameter search unit 27 uses the partial captured images 1 to K selected by the partial region selection unit 23 and the low resolution partial model images 1 to K created by the partial model image creation unit 26 to use the surface. The degree of fitness of the low-resolution partial model image with respect to the partial captured image is calculated for each area, and the top M parts having the highest fitness among the partial areas associated with the reflection parameters that are the basis of the low-resolution partial model image The region is determined as a candidate partial region, and the partial model image creation unit 26 is instructed to create subpixel partial model images for the top M candidate partial regions.

  Next, the partial model image creation unit 26 creates a subpixel partial model image having a resolution higher than the resolution of the partial captured image from the reflection parameters of the top M candidate partial regions, and the subpixel partial model image is a parameter search unit 27. Output to. The parameter search unit 27 calculates the degree of matching of the subpixel partial model image with respect to the partial captured image, and finally uses the reflection parameter of the partial area corresponding to the subpixel partial model image with the highest degree of matching as the reflection parameter of each surface area. To decide.

  Here, the calculation method of the above-mentioned fitness will be described in detail below. First, in order to compare the material data in the material database 25 with the subject image (partially photographed image), conversion processing according to the subject image photographing condition is performed as follows.

First, the luminance value observed from the camera direction is calculated for each of the R, G, and B channels according to the illumination conditions of the shooting environment. Various reflection models have been devised, but in this embodiment, the luminance value is calculated using a Phong reflection model, which is a parametric model often used in CG rendering. According to the Phong reflection model, the angle between the incident direction of the light source and the surface normal is γ, the angle between the viewpoint (camera) direction and the reflection vector of the light source is γ, the incident light intensity is I _i , and the diffuse reflectance is k. _{When d} and the specular reflectance are k _s , the observed luminance is expressed by the following equation.

If the brightness (incident intensity) of the light source for each channel is [r _i , g _i , b _i ], the observed color can be calculated by the following equation.

  Here, since γ depends on the viewpoint direction, the second term on the right side of the above equation is calculated for each of the video cameras 11 to 1K.

  Next, assuming that the material data of interest corresponds to a specific minute plane on the surface of the three-dimensional shape obtained by the visual volume intersection method, the coordinate system of the material data (minute plane) and each video camera 11 to 11 are assumed. Consider the geometric transformation between the coordinate systems of captured images observed at 1K.

In general, when a plane is photographed by a camera, the relationship between the coordinates on the plane as a subject and the coordinates of the photographed image can be expressed by a 3 × 3 matrix called a plane projective transformation matrix. If the minute area coordinates on the surface of the three-dimensional shape of interest are (x, y) and the image plane coordinates (x _k , y _k ) of the video camera k, the following relationship is obtained.

Here, H _k is a plane projective transformation matrix for the video camera k, and can be determined from the camera parameters of the video camera k and the position and orientation of the minute plane.

  Next, since the arrangement direction of the material data (the amount of rotation around the normal of the minute surface) is unknown, the following rotation conversion is considered. Here, R indicates rotation around the normal axis.

  From the above equation (4), the following equation is obtained.

Here, the actual observation image is further subjected to a reduction in resolution due to the characteristics of the optical system of the video camera. The observed luminance I _{c, (xk, yk) at} the coordinates (x _k , y _k ) on the video camera k regarding the color channel c (cε {R, G, B}) can be calculated by the following equation. Here, h (x _k , y _k , x, y, θ) is a deterioration function, and is approximated by a Gaussian distribution as follows.

Next, let I _k ^(j) (k = 1,..., K) be an image obtained by converting the material data j of interest for each of the K video cameras according to the above procedure, and the observed image in each video camera is J _k ( If k = 1,..., K), the compatibility r between them can be evaluated by the following equation.

  Next, with respect to the top M material data having the minimum fitness r (high fitness), the fitness is further determined with sub-pixel accuracy by the following equation, and is finally selected.

  An example of the reflection parameter search process in step S17 based on the above calculation method is schematically shown in the flowchart shown in FIG. Note that the flowchart shown in FIG. 3 expresses the flow of each process explicitly. In the above calculation method, each process is processed as a unit, and each process shown in FIG. The order of processing is not particularly limited to the illustrated example, and various changes can be made.

  First, the partial model image creation unit 26 reads material data (reflection parameters of each partial region) from the material database 25 in process 31, and in process 32, the material data is subjected to the material data according to Expression (6) and Expression (7). The two-dimensional rotation conversion process is executed.

  Next, in processing 33, the partial model image creation unit 26 determines the illumination conditions of the subject image input via the input unit 24 and the three-dimensional position and orientation of the surface region i calculated by the partial region selection unit 23. Then, the diffuse reflectance of the material data subjected to the two-dimensional rotation conversion is calculated according to the equations (2) and (8).

  Next, in order to create the partial model images 1 to K for the respective video cameras, that is, the partial captured images 1 to K, in the process 34, the partial model image creation unit 26 uses the illumination conditions, the three-dimensional position and the posture. Then, according to the formula (2) and the formula (8), the specular reflectance of the material data subjected to the two-dimensional rotation conversion is calculated. For example, the specular reflection parameter shown in FIG. 7 is calculated.

  Next, the partial model image creation unit 26 executes a planar projective transformation process on the material data in the process 35 using the three-dimensional position and orientation in accordance with the expressions (6) and (7), In 36, a resolution conversion process is performed on the material data in order to make the resolution of the material data coincide with the resolution of the partially captured image. In process 37, partial model images 1 to K are created.

  Next, in a process 36, the partial model image creation unit 26 compares the partial captured images 1 to K with the partial model images 1 to K to calculate the above-described fitness (Score j). After that, the top M material data with the minimum fitness is basically repeated in the same manner as described above, thereby obtaining the fitness with sub-pixel accuracy and having the smallest fitness (highest fitness). The reflection parameter is determined as the reflection parameter of the surface region i.

  Referring to FIG. 2 again, next, in step S18, the texture update unit 28 reads out the 3D model from the 3D model creation unit 22, and uses the Back Projection method to determine the surface area i determined by the parameter search unit 27. The texture of the 3D model is updated using the reflection parameter.

  Next, in step S19, the texture update unit 28 determines whether or not the above processing has been completed for all the surface regions. If the processing has not been completed for all the surface regions, The partial region selection unit 23 is instructed to start the region processing, and the processing from step S13 is repeated, and when the processing for all the surface regions is completed, the processing proceeds to step S20.

  Next, the CG creation unit 29 creates a CG image using a three-dimensional model with an updated texture as a component to be placed in a virtual scene in the CG environment, and then processes the next captured image. The process after step S11 is repeated. For example, the texture of the partially captured image shown in FIG. 8 is updated, and the CG image shown in FIG. 9 is created. FIG. 10A shows an example of a CG image before the texture update, and FIG. 10B shows an example of the CG image after the texture update. It can be seen that the definition is high.

  Through the above processing, in the present embodiment, a plurality of subject images obtained by photographing subjects from a plurality of different camera viewpoints are acquired, a three-dimensional model of the subject is created from the plurality of subject images, and the surface of the three-dimensional model Are divided into a plurality of surface areas, and a partial captured image of a corresponding area corresponding to each surface area is selected for each subject image.

  On the other hand, in the material database 25, material data including a specular reflection parameter and a diffuse reflection parameter is stored in advance for each partial area obtained by dividing the surface of the subject into a plurality of parts. The diffuse reflection parameters are read out, and a partial model image corresponding to each partial captured image is created using these specular reflection parameters, diffuse reflection parameters, and illumination conditions in which a plurality of subject images are captured. Thereafter, the suitability of the partial model image with respect to the partially captured image is calculated for each surface area, and the reflection parameter of the partial area corresponding to the partial model image having the highest fitness is determined as the reflection parameter of each surface area. The specular reflection parameter and the diffuse reflection parameter close to the real object can be dynamically determined as the specular reflection parameter and the diffuse reflection parameter of the model surface, and the texture can be dynamically updated.

  In this embodiment, two-dimensional data is stored as material data, and optimum material data is searched from all the material data. However, the material data is three-dimensionally configured and each surface of the three-dimensional model is stored. Material data close to the region may be selected in advance, and optimum material data may be searched from material data narrowed down to some extent. Further, although the specular reflection parameter is constant in the minute region R, different specular reflection parameters may be stored depending on the position in the minute region R as in the case of the diffuse reflection parameter.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart for demonstrating the reflection parameter update process by the image processing apparatus shown in FIG. It is a figure which shows typically the reflection parameter search process shown in FIG. It is a figure which shows an example of material data. It is a figure which shows an example of a to-be-photographed image. It is a figure which shows an example of a three-dimensional model. It is a figure which shows an example of a specular reflection parameter. It is a figure which shows an example of a partial picked-up image. It is a figure which shows an example of the CG image in which the texture was updated. It is a figure which shows an example of the CG image in which the detail was highly defined by the presence or absence of the update of a texture.

Explanation of symbols

11 to 1K video camera 21 image acquisition unit 22 three-dimensional model creation unit 23 partial region selection unit 24 input unit 25 material database 26 partial model image creation unit 27 parameter search unit 28 texture update unit 29 CG creation unit

Claims (7)

An image processing apparatus for determining a reflection parameter of a surface of a subject model,
A database unit that stores a reflection parameter in advance for each partial region obtained by dividing the surface of the subject into a plurality of parts;
An acquisition unit that acquires a plurality of subject images obtained by photographing the subject from a plurality of different camera viewpoints;
A model creation unit for creating a three-dimensional model of a subject from the plurality of subject images;
A selection unit that divides the surface of the three-dimensional model into a plurality of surface regions and selects a partial captured image of a corresponding region corresponding to each surface region for each of the subject images;
An image creation unit that reads reflection parameters of each partial region from the database unit and creates partial model images corresponding to the partial captured images using the read reflection parameters and illumination conditions under which the plurality of subject images are captured. When,
A determination unit that calculates a degree of matching of the partial model image with respect to the partial captured image for each surface area, and determines a reflection parameter of the partial area corresponding to the partial model image having the highest degree of matching as a reflection parameter of each surface area; An image processing apparatus comprising: The database unit stores in advance reflection parameters calculated from an image of the subject photographed at a first resolution,
The obtaining unit obtains a plurality of subject images obtained by photographing the subject at a second resolution lower than the first resolution;
The image processing apparatus according to claim 1, wherein the image creation unit creates a low-resolution partial model image obtained by converting the resolution from the first resolution to the second resolution as the partial model image. The determining unit calculates a degree of matching of the low-resolution partial model image with respect to the partial captured image, and determines a predetermined number of partial areas having a high degree of matching as candidate partial areas;
The image creating unit creates a sub-pixel partial model image having a third resolution higher than the second resolution from the reflection parameter of the candidate partial region;
The determining unit calculates a degree of matching of the subpixel partial model image with respect to the partial captured image, and determines a reflection parameter of a partial region corresponding to the subpixel partial model image having the highest degree of matching as a reflection parameter of each surface region. The image processing apparatus according to claim 2, wherein:   The image processing apparatus according to claim 1, wherein the database unit stores in advance a diffuse reflection parameter and a specular reflection parameter as the reflection parameter. The database unit stores in advance the specular reflection parameter for each partial region as a constant specular reflection parameter in the partial region,
The image processing apparatus according to claim 4, wherein the image creation unit creates a partial model image corresponding to each partial captured image with a specular reflection parameter in the partial region being constant. An image processing apparatus including a database unit that stores reflection parameters in advance for each partial region obtained by dividing the surface of a subject, an acquisition unit, a model creation unit, a selection unit, an image creation unit, and a determination unit An image processing method for determining a reflection parameter of a surface of a subject model using
The obtaining unit obtaining a plurality of subject images obtained by photographing the subject from a plurality of different camera viewpoints;
The model creating unit creating a three-dimensional model of a subject from the plurality of subject images;
The selection unit dividing the surface of the three-dimensional model into a plurality of surface regions, and selecting a partial captured image of a corresponding region corresponding to each surface region for each subject image;
The image creation unit reads reflection parameters of each partial region from the database unit, and uses the read reflection parameters and the illumination conditions under which the plurality of subject images are captured, and the partial model image corresponding to each partial captured image The steps of creating
The determination unit calculates the degree of matching of the partial model image with respect to the partial captured image for each surface area, and the reflection parameter of the partial area corresponding to the partial model image having the highest degree of matching is used as the reflection parameter of each surface area. And a determining step. An image processing program for determining a reflection parameter of a surface of a subject model,
A database unit that stores a reflection parameter in advance for each partial region obtained by dividing the surface of the subject into a plurality of parts;
An acquisition unit that acquires a plurality of subject images obtained by photographing the subject from a plurality of different camera viewpoints;
A model creation unit for creating a three-dimensional model of a subject from the plurality of subject images;
A selection unit that divides the surface of the three-dimensional model into a plurality of surface regions and selects a partial captured image of a corresponding region corresponding to each surface region for each of the subject images;
An image creation unit that reads reflection parameters of each partial region from the database unit and creates partial model images corresponding to the partial captured images using the read reflection parameters and illumination conditions under which the plurality of subject images are captured. When,
As a determination unit that calculates the degree of matching of the partial model image with respect to the partial captured image for each surface area, and determines the reflection parameter of the partial area corresponding to the partial model image with the highest degree of matching as the reflection parameter of each surface area An image processing program for causing a computer to function.