JP2020166652A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To suppress misalignment of texture between polygons in three-dimentional polygon data with which a texture image is associated.SOLUTION: An image processing apparatus 100 that generates a texture image to be associated with three-dimensional polygon data of an object, determines a pixel value of each of a plurality of pixels structuring a region corresponding to polygons constituting the three-dimensional polygon data on the basis of one or more captured image on a per-pixel basis, and generates a texture image on the basis of the pixel values.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing apparatus, an image processing method and a program.

There is known a technique for reconstructing an image (virtual viewpoint image) obtained when the subject is viewed from an arbitrary viewpoint from a plurality of images obtained by capturing the subject with a plurality of cameras. At that time, three-dimensional data in which a texture image is associated with each constituent element is used.

Patent Document 1 discloses a method for generating three-dimensional polygon data, which is an example of the three-dimensional data. In Patent Document 1, an captured image is selected for each polygon constituting the three-dimensional polygon data, and three-dimensional data associated with a texture image based on the selected captured image is generated.

Japanese Unexamined Patent Publication No. 2003-337953

However, when the captured image is selected for each polygon, the selected image may differ greatly between the adjacent polygons, and the texture shift between the polygons occurs.

Therefore, an object of the present invention is to suppress the deviation of the texture between polygons in the three-dimensional polygon data to which the texture image is associated.

The image processing apparatus of one aspect of the present invention generates a texture image corresponding to the three-dimensional polygon data of the object based on a plurality of captured images acquired by imaging the object using a plurality of imaging devices. In an image processing device, the pixel values of a plurality of pixels or a plurality of pixel groups constituting a region corresponding to a polygon constituting the three-dimensional polygon data are captured by the plurality of pixels or the plurality of image pickups for each of the plurality of pixel groups. It has a determination means for determining based on one or more captured images of the image, and a generation means for generating the texture image based on the pixel value determined by the determination means.

According to the present invention, it is possible to suppress the deviation of the texture between polygons in the three-dimensional polygon data to which the texture image is associated.

The block diagram which shows the hardware configuration example of an image processing apparatus. Conceptual diagram of virtual viewpoint image generation. A conceptual diagram of the correspondence between the three-dimensional shape data and the texture data. An example of data that represents 3D polygon data associated with a texture image. A conceptual diagram showing the front direction of a polygon. The figure which shows the configuration example of the system used for the generation process of the texture image of Embodiment 1. FIG. The block diagram which shows the functional structure example of the image processing apparatus in Embodiment 1. FIG. The flowchart which shows the example of the image processing method in Embodiment 1. The block diagram which shows the functional structure example of the image processing apparatus in Embodiment 2. The flowchart which shows the example of the image processing method in Embodiment 2. The figure explaining the concept of the texture image generation processing of Embodiment 2.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

In the present embodiment, a model in which textures are associated with elements constituting 3D data is expressed as a 3D model. The three-dimensional shape data (sometimes referred to simply as shape data) is data representing the three-dimensional shape of an object (subject), and is typically represented by polygon data. Here, the case where the three-dimensional shape data and the shape representation of the three-dimensional model are performed using the polygon data will be described. The element of the polygon data is a polygon, and the polygon is defined by a plurality of vertices. The polygon is not limited to a triangle, and may be a quadrangle or a polygon larger than that. In particular, in the following, a three-dimensional model in which the shape is expressed using triangular polygons will be described as an example.

The texture includes at least one of color information, luminance information, and saturation information. Further, the texture image is an image having a plurality of texture areas, and each texture area is associated with any polygon. The texture area contains at least one of color information, luminance information, and saturation information related to the corresponding polygon.

FIG. 2 shows a conceptual diagram of generating a virtual viewpoint image. FIG. 2A represents the shape data of the subject, and FIG. 2B represents the texture image of the subject. The texture image is composed of a plurality of texture areas corresponding to a polygon or a region composed of a plurality of polygons. The pixel value of the texture area represents the texture of the corresponding polygon or the area composed of a plurality of polygons. The texture image shown in FIG. 2B has two texture regions, and each of the texture regions corresponds to each of the two hexagons in FIG. 2A. By combining FIGS. 2A and 2B and performing rendering processing based on the determined viewpoint, a virtual viewpoint image corresponding to the determined viewpoint as shown in FIG. 2C is generated. .. The viewpoint may be determined by the user or may be automatically determined by the device. Further, the determination of the viewpoint may include at least the determination of the position and posture (direction) of the viewpoint.

Next, an expression method for associating FIGS. 2A and 2B will be described with reference to FIG. This is equivalent to generating a three-dimensional model (hereinafter referred to as a 3D polygon model) by associating texture data with each polygon of shape data (hereinafter referred to as 3D polygon data) composed of a plurality of triangular polygons. To do. A 3D polygon model in which texture data is associated with each polygon may be referred to as a textured 3D polygon model.

FIG. 3A explicitly represents polygons T0 to T11, which are elements for expressing the shape data of the subject shown in FIG. 2A, and vertices V0 to V11 constituting them. is there. Further, FIG. 3B represents the texture image shown in FIG. 2B and the positions P0 to P13 corresponding to the respective vertices of the polygon in FIG. 3A. Then, as information for associating these, as shown in FIG. 3C, the IDs of the polygon vertices in the three-dimensional space and the IDs of the texture vertices in the texture image space that constitute them for each polygon A correspondence table is generated. As a result, the polygon can be textured.

The coordinates of FIG. 3 (A) are represented by the three-dimensional space coordinates of the xyz axis, and the coordinates of FIG. 3 (B) are represented by the 2D image space coordinates of the uv axis. As shown in polygon vertices V0 to V4 and V7 to V11 in FIG. 3C, in many cases, the polygon vertices and the texture vertices have a one-to-one correspondence, and the index numbers can be matched and expressed. However, for example, as the polygon vertex V5 corresponds to both the texture vertices P5 and P12, even if it is one polygon vertex in the three-dimensional space, it corresponds to a plurality of different positions in the texture image space. There are texture vertices. The vertex ID and the texture vertex ID are managed independently so that the virtual viewpoint image generation process can be performed even in such a correspondence relationship.

FIG. 4 is a data representation of a textured 3D polygon. Specifically, FIG. 4A is data showing the correspondence between the polygon vertices represented by FIG. 3A and their coordinates. FIG. 4B is data showing the correspondence between the texture vertices represented by FIG. 3B and their coordinates. FIG. 4C is the same as FIG. 3C, and is data showing the correspondence between the triangular polygon, the polygon vertices of the triangular polygon, and the texture vertices. FIG. 4 (D) is a texture image represented by FIG. 2 (B) (or FIG. 3 (B)).

Further, the order of the polygon vertex IDs in FIG. 4A has a role of defining the front direction of the surface. For example, polygon T0 is composed of three vertices, and there are six orders. The direction that follows the right-handed screw rule with respect to the direction of rotation when the vertices are traced in order from the left is often defined as the front direction. 5 (A) and 5 (B) show the description order of polygon vertices and the set of polygons in the front direction. In FIG. 5A, the direction from the back to the front with respect to the paper surface is the front direction, and in FIG. 5B, the direction from the front to the back with respect to the paper surface is the front direction. It can be applied to this embodiment in any direction.

In this embodiment, for example, a polygon representation of a quadrangle or more may be used without being bound by the data representation described above. Furthermore, this embodiment can be applied to various cases such as when the coordinates are directly described without using an index to express the correspondence between the shape and the texture.

<Embodiment 1>
Regarding this embodiment, first, an outline of the texture image generation process will be described with reference to FIG. FIG. 6 shows a configuration example of a system used for a texture image generation process. In the present embodiment, the texture image is generated in roughly two steps.

In the first step, the coordinates on the texture image 608 are determined for each polygon in the 3D polygon model of the subject. First, 3D polygon data of the subject 602 is generated based on an image group obtained by capturing the subject 602 from a plurality of different imaging devices (viewpoints) 601. Then, the captured image 604 of the imaging device that captures the polygon 603 of interest in the 3D polygon data in a state close to the front with high resolution is selected. Next, the pixel region 607 corresponding to the polygon of interest 603 is detected in the selected captured image 604, the texture region 609 corresponding to the pixel region 607 is arranged at an arbitrary position of the texture image 608, and the coordinates thereof are determined. .. As a result, on the texture image 608, the texture area 609 showing the texture of the polygon 603 of interest of the 3D polygon data of the subject is associated and secured.

However, the captured image including the region on the subject corresponding to the polygon of interest 603 is other than the captured image 604, for example, the captured image 605. For example, suppose that the captured image 605 is selected and the coordinates on the texture image 608 are determined. The imaging direction of the imaging device 601 that acquires the captured image 605 is a larger angle than the imaging direction of the imaging device 601 that acquires the captured image 604 with respect to the direction facing the region on the subject corresponding to the polygon of interest 603. Is the direction to have. Further, the image pickup device 601 that acquires the captured image 605 captures a wide-angle range as compared with the image pickup device 601 that acquires the captured image 604. Therefore, the area corresponding to the polygon of interest 603 on the captured image 605 is smaller than the area corresponding to the polygon of interest 603 on the captured image 604. Therefore, when the captured image 605 is selected and the coordinates on the texture image 608 are determined, the texture area becomes smaller than when the captured image 604 is selected and the coordinates on the texture image 608 are determined. As a result, the texture area corresponding to the polygon of interest 603 has a low resolution. Here, it is assumed that the resolution regarding the texture is defined by the number of pixels.

Based on the above points, in the present embodiment, the captured image obtained by capturing the polygon of interest with high resolution and close to the front is selected from the plurality of captured images, and the coordinates of the texture image 608 are determined. To secure the texture area 609. As a result, it is possible to generate a high-resolution textured 3D polygon model. The texture area 609 secured on the texture image 608 has a size equivalent to that of the pixel area 607 and is a triangular area, but is not limited to this. As a rectangular region including the pixel region 607, another method of securing the texture region 609 on the texture image 608 may be used.

On the other hand, as a method for generating a 3D polygon model, a known method can be used. However, the generated 3D polygon model may contain an error. In particular, when there is a long distance between the subject and the imaging device, such as when multiple imaging devices are placed in the stadium and a 3D polygon model of a player on the ground is generated, the error becomes large. .. If the 3D polygon model contains an error, the polygon of interest and the texture associated with that polygon may differ from the actual texture. This phenomenon becomes a problem when generating a texture image in a 3D polygon model. For example, it is assumed that the pixel value of the texture image is uniformly determined by using only the imaging device selected in the first step. Then, for the polygon adjacent to the polygon of interest 603, when the corresponding pixel area of the image captured by the imaging device different from the polygon of interest 603 is selected, the polygons adjacent to each other are affected by the error included in the 3D polygon data. Texture shift occurs. This deviation is not noticeable when the texture is uniform, but when the subject is a person, unnatural cuts occur in the face and uniform number part, which is a factor of image quality deterioration. It becomes. In this way, when the same captured image is used for each polygon, it is possible to generate a high-resolution texture image, but the image quality deteriorates due to the texture shift between the polygons due to the influence of the error included in the 3D polygon data. ..

Therefore, in the present embodiment, the following second step is performed. In this second step, the pixel values of the plurality of pixels included in the texture region 609 corresponding to the polygon of interest 603 on the texture image 608 secured in the first step are set to the captured images of the plurality of different imaging devices 601. Determined based on. Specifically, first, for each pixel constituting the texture region 609 (or texture image 608), the resolution on the captured image, the degree of front of the imaging device with respect to the normal direction of the subject 602 corresponding to the pixel of interest, and the like are determined. The evaluation value is calculated based on this. Then, the pixel values of the captured images are combined according to the ratio of each of the two or more captured images of the two or more imaging devices determined based on the calculated evaluation value, and the pixel values of the captured images are combined on the texture region 609 (or texture image 608). Determine the pixel value. By determining an appropriate ratio for each pixel of the texture region 609 (or texture image 608) in this way, it is possible to reduce image quality deterioration due to texture shift between polygons.

That is, in the present embodiment, the texture area for ensuring the resolution of the texture image is secured in the first step. Then, in the next second step, the texture between polygons is determined by combining two or more captured images for each pixel included in the high-resolution texture region at an appropriate ratio and determining the pixel value. It is possible to reduce the deterioration of image quality due to the deviation. As a result, it is possible to generate a high-quality textured 3D polygon model in which there is no texture shift between polygons and the texture resolution is high.

In the second step, it is not necessary to determine the pixel value for each pixel constituting the texture region 609 based on two or more captured images. For example, the pixel value may be determined based on two or more captured images for each pixel group composed of a plurality of pixels constituting the texture region 609. In this case, the pixel group is an area obtained by dividing the texture area 609 into several areas, and the number of pixels of the pixel group is smaller than the number of pixels of the texture area 609. Further, the pixel value may be determined based on one captured image for each pixel or pixel group constituting the texture region 609. Also in this case, since the determined pixel value is determined not in polygon units but in units smaller than that, the deviation of the texture between adjacent polygons is reduced. That is, one or more captured images may be selected for each pixel or pixel group constituting the texture region 609, and the pixel value of the pixel or pixel group may be determined based on the selected one or more captured images.

The above is the outline of the texture image generation process performed in the present embodiment. In this embodiment, a case where a person such as a player is imaged in a stadium-scale environment will be described as an example, but the present embodiment is not limited to this, and may be used in a small-scale imaging environment such as when an arbitrary subject is imaged in a studio or the like. Applicable.

Hereinafter, a specific configuration of the present embodiment will be described. FIG. 1 is a diagram showing an example of a hardware configuration of the image processing device of the present embodiment. The image processing device 100 of the present embodiment includes a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface 105, and an output interface 106. Then, each component of the image processing apparatus 100 is connected to each other by a system bus 107. Further, the image processing device 100 is connected to the external storage device 108 via the input interface 105, and is connected to the external storage device 108 and the display device 109 via the output interface 106.

The CPU 101 is a processor that uses the RAM 102 as a work memory to execute a program stored in the ROM 103 and collectively controls each component of the image processing device 100 via the system bus 107. As a result, various processes described later are executed.

The secondary storage device 104 is a storage device that stores various data handled by the image processing device 100, and an HDD is used in this embodiment. The CPU 101 can write data to the secondary storage device 104 and read data stored in the secondary storage device 104 via the system bus 107. In addition to the HDD, various storage devices such as an optical disk drive and a flash memory can be used for the secondary storage device 104.

The input interface 105 is, for example, a serial bus interface such as USB or IEEE1394, and input of data, commands, etc. from an external device to the image processing device 100 is performed via the input interface 105. The image processing device 100 acquires data from an external storage device 108 (for example, a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory) via the input interface 105. An input device such as a mouse or a button (not shown) can also be connected to the input interface 105. The output interface 106 includes a serial bus interface such as USB or IEEE 1394, similarly to the input interface 105. In addition, for example, a video output terminal such as DVI or HDMI (registered trademark) can be used. The output of data or the like from the image processing device 100 to the external device is performed via the output interface 106. The image processing device 100 displays an image by outputting the processed image or the like to a display device 109 (various image display devices such as a liquid crystal display) via the output interface 106. Although there are other components of the image processing apparatus 100 other than the above, the description thereof will be omitted because they are not the main focus of the present invention.

Hereinafter, the processing performed by the image processing apparatus 100 of the present embodiment will be described with reference to FIGS. 7 and 8. FIG. 7 is a block diagram showing a functional configuration of the image processing device 100. FIG. 8 shows a flowchart of an image processing method performed by the image processing apparatus 100 in the present embodiment. As shown in FIG. 7, the image processing apparatus 100 includes an image data acquisition unit 701, a viewpoint information acquisition unit 702, a 3D polygon acquisition unit 703, a UV coordinate acquisition unit 704, a first calculation unit 705, an area determination unit 706, and a second. It has a calculation unit 707 and an image generation unit 708. The image processing device 100 functions as each component shown in FIG. 7 by executing the program stored in the ROM 103 by the CPU 101 using the RAM 102 as the work memory, and executes a series of processes shown in the flowchart of FIG. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and even if the image processing device 100 is configured so that a part or all of the processes is performed by one or a plurality of processing circuits other than the CPU 101. Good. Hereinafter, the flow of processing performed by each component will be described.

In step S801, the image data acquisition unit 701 acquires a plurality of captured images obtained by capturing the subject from a plurality of different viewpoints via the input interface 105 or from the secondary storage device 104. The captured image is, for example, an image captured by a plurality of imaging devices 601 arranged so as to surround the subject 602 in FIG. In the present embodiment, as shown in FIG. 6, the image pickup apparatus 601 is arranged so as to look down on the floor surface from above. Further, a different zoom lens is attached to each of the image pickup devices 601, and the size of the subject captured on the captured image such as the captured image 604 and the captured image 605 may be different for each imaging device. The method of arranging the image pickup device 601 and the configuration of the zoom lens to be attached shown in FIG. 6 are merely examples, and a plurality of captured images may be acquired using other configurations.

Further, the image processing device 100 may be connected to a plurality of image processing devices and may be configured as a part of an image processing system including the image processing device 100 and the plurality of image processing devices. According to such a configuration, it is possible to generate a textured 3D polygon model while imaging the subject. In the present embodiment, the captured image to be acquired will be described by taking the case of a color image of RGB3 channel as an example. However, this embodiment can be applied even when the captured image is a 1-channel gray image or moving image data. When the captured image is moving image data, the image processing device 100 can perform the following processing using frames captured by a plurality of imaging devices at substantially the same time. Further, the image data acquisition unit 701 stores each image in association with a number for distinguishing the image pickup device 601 (hereinafter, referred to as a viewpoint number) in order to distinguish the captured images. The image data acquisition unit 701 outputs the captured image to the image generation unit 708.

The image data acquisition unit 701 may acquire a subject image which is a region representing a subject in the captured image instead of the captured image, or may acquire both of them. The subject image may be generated by extracting a region of a moving subject from the captured image by using a known foreground background separation method. In particular, it may be a subject image generated in the process of generating three-dimensional shape data based on a plurality of captured images. Further, the subject image may be generated based on the silhouette image showing the silhouette of the subject and the captured image. Further, the image data acquisition unit 701 may generate a subject image from the captured image. However, the subject image has information on the texture of the subject.

In step S802, the viewpoint information acquisition unit 702 acquires the imaging viewpoint information of the plurality of imaging devices that have captured the plurality of captured images acquired by the image data acquisition unit 701 for each imaging device. The imaging viewpoint refers to the viewpoint of the imaging device 601 and the imaging viewpoint information refers to information about each of the plurality of imaging devices 601. The imaging viewpoint information includes the position / orientation information of the imaging device 601 within a predetermined coordinate system, and includes, for example, the position information of the imaging device 601 and the attitude information indicating the optical axis direction. Further, the image pickup viewpoint information may include the angle of view information of the image pickup device 601 such as the focal length or the principal point position of the image pickup device 601. Using these imaging viewpoint information, it is possible to associate each pixel of the captured image with the position of the subject existing in the captured image. Therefore, it is possible to identify the corresponding pixel on the captured image and acquire the color information of the specific portion of the subject. Further, the imaging viewpoint information can include a distortion parameter indicating distortion of the captured image acquired by the imaging device 601 and an F value, and imaging parameters such as shutter speed and white balance. The viewpoint information acquisition unit 702 outputs the imaging viewpoint information to the uv coordinate acquisition unit 704, the first calculation unit 705, the area determination unit 706, and the second calculation unit 707.

In step S803, the 3D polygon acquisition unit 703 acquires 3D polygon data in which the shape of the subject in the three-dimensional space is represented by a triangular polygon. The 3D polygon data is represented by the data format shown in FIG. 4 (A). The 3D polygon acquisition unit 703 applies the visual hull algorithm to acquire voxel information and reconstructs the 3D polygon data. The 3D polygon acquisition unit 703 may acquire 3D polygon data that has been generated in advance and stored in the secondary storage device 104 or the external storage device 108.

The method for generating 3D polygon data may be arbitrary, and for example, voxel information may be directly converted into polygon data. In addition, PSR (poisson surface reconstruction) may be applied to a point cloud obtained from a depth map acquired by using an infrared sensor. The point cloud acquisition method may be obtained by stereo matching using image features typified by PMVS (Patch-based Multi-view Stereo). The 3D polygon acquisition unit 703 outputs the acquired 3D polygon model to the uv coordinate acquisition unit 704, the first calculation unit 705, the area determination unit 706, and the second calculation unit 707.

In step S804, the uv coordinate acquisition unit 704 uses the acquired imaging viewpoint information to project the polygon vertices of the acquired 3D polygon model onto the respective imaging viewpoints, and the uv on the captured image of the imaging viewpoint. Calculate and obtain the coordinates. That is, the uv coordinates on all the captured images corresponding to each polygon vertex are acquired. However, when it is found that the polygon vertices are outside the angle of view of the imaging device representing the imaging viewpoint, a negative value is set as an error value in the uv coordinates. This negative value can be used as a flag for disabling the imaging viewpoint that is out of the angle of view in the subsequent processing. However, the method of expressing the error value is not limited to the above, and an arbitrary value that is within the angle of view and can be distinguished from the acquired uv coordinate value may be set. The uv coordinate acquisition unit 704 calculates the uv coordinate values for all the imaging viewpoints for all the polygon vertices included in the 3D polygon data, and outputs the uv coordinate data to the first calculation unit 705.

In step S805, the first calculation unit 705 acquires the evaluation value Q at each imaging viewpoint from the viewpoint information acquisition unit 702 for all the polygons included in the 3D polygon data acquired from the 3D polygon acquisition unit 703. Calculate using information. The calculated evaluation value Q is used when performing the processing of the first step of generating the texture image described above. Specifically, it serves as an index for selecting one imaging device (captured image) from a plurality of imaging devices (captured images) when determining the coordinates of the texture region in the texture image. The evaluation value Q is calculated for each polygon for all imaging viewpoints. The first calculation unit 705 outputs the calculated evaluation value Q to the area determination unit 706. The method of calculating the evaluation value Q in this embodiment will be described below.

(Calculation method of evaluation value Q)
The method of calculating the evaluation value Q will be described in detail. Hereinafter, a method of calculating an evaluation value when focusing on one polygon and one imaging viewpoint will be described. The same process is applied to all the polygons included in the 3D polygon for the number of imaging viewpoints, and the evaluation values of all the polygons are calculated. Since the evaluation value Q is calculated for each imaging viewpoint, the evaluation value for the imaging viewpoint acquired by the viewpoint information acquisition unit 702 is held for each polygon. In the following, the focused imaging viewpoint will be referred to as the focused viewpoint.

First, in the uv coordinate data acquired from the uv coordinate acquisition unit 704, it is determined whether or not all the polygon vertices constituting the polygon are included in the angle of view of the viewpoint of interest. As described above, when it is outside the angle of view, a negative value is held as an error value in the UV coordinates, so it is confirmed that the UV coordinates of all the polygon vertices are positive. If the uv coordinates of all polygon vertices are positive, the calculation process of the evaluation value Q proceeds. On the other hand, if the result of the determination is negative, the evaluation value Q is set to a negative value. However, the evaluation value Q to be set when the determination result is negative is not limited to a negative value, and an arbitrary numerical value that can be distinguished from the result calculated by calculating the evaluation value Q may be set.

Next, as representative points of the polygon of interest, the center of gravity C of the three vertices constituting the polygon and the normal direction vector N of the polygon are calculated. Further, the direction vector CA of the viewpoint of interest is calculated from the position coordinates of the viewpoint of interest in the three-dimensional space and the center of gravity C. By calculating the inner product of the calculated normal direction vector N and the direction vector CA of the viewpoint of interest, the cosine cos θ of the angle θ formed by the two direction vectors is calculated. Here, the normal direction is the direction of the table that is perpendicular to the polygon plane and is defined by the order of the polygon vertices. The direction of the viewpoint of interest is the direction from the center of gravity C of the polygon toward the viewpoint of interest defined by the external parameters. The external parameters are the position information of the imaging device and the attitude information indicating the direction of the optical axis.

When the calculated cos θ is a negative value, it means that the surface of the polygon is not reflected in the captured image of the viewpoint of interest. Therefore, when cosθ is negative, the evaluation value Q is set to a negative value so that the captured image of the viewpoint of interest is not used. On the other hand, when cosθ is positive, the calculation process of the evaluation value Q proceeds.

Next, the resolution of the pixel region corresponding to the polygon in the captured image of the viewpoint of interest is calculated, and the evaluation value Q is calculated using the calculated cos θ. This resolution is defined here as the area (for example, the number of pixels) S of the triangle projected on the captured image of the viewpoint of interest. Then, the imaging viewpoint having a large resolution is preferentially selected. However, as described above, when the shape includes an error and the inclination with respect to the subject plane is large, the texture is distorted. Therefore, as the evaluation value Q, cosθ is used as the distortion index W in addition to the resolution. As a result, it is possible to select a high-resolution texture region after distinguishing between an imaging viewpoint in which a mapping with a large distortion is likely to be performed and an imaging viewpoint in which a mapping with a small distortion is likely to be performed. Summarizing the above, the evaluation value Q is described by the following equation 1.

Q = S × W ・ ・ ・ Equation 1
The definition of the resolution S may be another method as long as it is an index of the resolving power in the imaging device (imaging viewpoint) that images a partial region of the subject corresponding to the polygon. For example, the resolution S may be calculated by a mathematical formula or a look-up table from the focal length, the distance between the image pickup device and the partial region of the subject corresponding to the polygon, and the like. Further, the degree of distortion W of the texture may be other than cos θ, or the angle itself or a weight set based on the angle may be used. For example, the threshold value of the angle is set, and W = 1 when the angle is equal to or less than the threshold value, and W = 0 when the angle is larger than the threshold value. In that case, the weight does not necessarily have to be binary, it may be 3 or more, and the weight is set to decrease linearly or non-linearly from the angle 0 to the threshold angle. May be good. In the present embodiment, the degree of influence on the evaluation value Q is treated as the same for the resolution index S and the distortion index W, but the present invention is not limited to this. One of the indexes may be squared, and the degree of influence on the evaluation value Q may be set differently.

Returning to FIG. 8, in step S806, the area determination unit 706 determines the texture area corresponding to each polygon on the texture image based on the evaluation value Q for each polygon acquired from the first calculation unit 705. Specifically, the area determination unit 706 determines the coordinates on the texture image that defines the texture area. The area determination unit 706 selects an imaging viewpoint that maximizes the evaluation value Q calculated by the first calculation unit 705 for each polygon. Hereinafter, the selected imaging viewpoint is referred to as a selection viewpoint.

Next, the captured image of the selected viewpoint is projected onto the corresponding polygon, and the pixel area of the projected captured image of the selected viewpoint is secured on the texture image as a region corresponding to the texture of the polygon. Here, the uv coordinates on the texture image corresponding to the secured texture area are calculated so that the polygon can refer to the required position on the texture image, and the polygon and the texture area are in the format shown in FIG. 4 (C). Save the data associated with. Further, the area determination unit 706 has three pixels, that is, the identification ID of the corresponding polygon, the viewpoint number of the selected viewpoint, and the pixel coordinates (u, v) corresponding to the captured image of the selected viewpoint for each pixel in the secured texture area. Hold data. Based on this information, it is possible to take correspondence with polygons in the pixels included in the texture image and calculate the value of the three-dimensional coordinates corresponding to the pixels. The specific calculation method will be described later.

The texture areas for each polygon secured on the texture image may be arranged in any order and position on the texture image. When there is a pixel that does not correspond to any polygon included in the 3D polygon data among the pixels included in the texture image (for example, the white area in FIG. 2B), the identification ID of the polygon corresponding to the pixel Gives a dummy value to. For example, a negative value may be given as the identification ID.

The area determination unit 706 determines the arrangement of the texture area on the texture image for all polygons. After that, the area determination unit 706 uses the second calculation unit 707 and the image generation unit to generate a texture image in which the polygon identification ID for each pixel, the identification number of the selected viewpoint, and the pixel coordinates on the captured image of the selected viewpoint are held for each pixel. Output to 708.

In step S807, the second calculation unit 707 sets the pixel of interest for determining the pixel value in the pixels included in the texture image. Here, the upper left pixel in the texture image is first selected as the pixel of interest, and then each time the determination of the pixel value of the pixel of interest is completed (No in S810), the pixel of interest is set toward the lower right. Pixels that are not selected are selected as new pixels of interest. The selection order of the pixels of interest is not limited to this, and the pixels of interest may be determined in any order.

In step S808, the second calculation unit 707 calculates the evaluation value V of the pixel of interest for each imaging viewpoint using the texture image acquired from the region determination unit 706. The evaluation value V calculated here is used as an index when determining the pixel value of the texture in the pixel of interest. The second calculation unit 707 outputs the calculated evaluation value V at all viewpoints to the image generation unit 708 together with the index of the distortion of the texture used for calculating the evaluation value V and the pixel coordinates for each imaging viewpoint held. The calculation method of the evaluation value V will be described below.

(Calculation method of evaluation value V)
The method of calculating the evaluation value V in S808 will be described in detail. First, as a preliminary preparation for the calculation of the evaluation value V, the three-dimensional coordinates and the normal vector on the 3D polygon data corresponding to the pixel of interest are calculated. In the present embodiment, the polygon identification ID included in the texture image, the identification number of the selected viewpoint, and the pixel coordinates on the captured image of the selected viewpoint are used to calculate the three-dimensional coordinates corresponding to the pixel of interest. Specifically, the light beam is virtually emitted to the pixel coordinates on the captured image of the selected viewpoint, starting from the three-dimensional coordinates corresponding to the viewpoint position of the selected viewpoint acquired from the imaged viewpoint information in the viewpoint information acquisition unit 702. Then, the point at which the polygon corresponding to the polygon identification ID (hereinafter referred to as the polygon of interest) collides with the above-mentioned light ray is set as the three-dimensional coordinates corresponding to the pixel of interest. Further, the normal vector of the pixel of interest is calculated from the three-dimensional coordinates corresponding to the pixel of interest, the three-dimensional coordinates of the three vertices of the polygon of interest, and the normal vector of the three vertices. Specifically, based on the respective distances between the 3D coordinates corresponding to the pixel of interest and the 3D coordinates of the 3 vertices of the polygon of interest, the 3D corresponding to the pixel of interest is determined by the weighted sum of the normal vectors at each vertex. Calculate the normal vector of the coordinates.

Next, the evaluation value V is calculated. Hereinafter, the method of calculating the evaluation value V of one imaging viewpoint at a plurality of imaging viewpoints will be described, but the evaluation value V is calculated by performing the same processing for the other imaging viewpoints. In the following, one imaging viewpoint will be described as a viewpoint of interest.

First, starting from the three-dimensional coordinates corresponding to the pixel of interest calculated as described above, a ray is virtually shot in the direction of the normal vector, and it is determined whether or not the ray collides with the angle of view of the viewpoint of interest. .. If the result of the determination is negative, the three-dimensional coordinates are not visible from the viewpoint of interest, so an error value is set in the evaluation value V in order to exclude the pixel of interest in the calculation of the pixel value of the pixel of interest. In the present embodiment, a negative value is used as the error value, but the error value is not limited to this. If the determination result is acceptable, the coordinates of the colliding pixel are retained and the evaluation value V in the pixel of interest is calculated.

In the present embodiment, the evaluation value V = S × W is calculated by using the resolution index S and the texture distortion index W in the same manner as the concept of the evaluation value Q described in step S805. However, as the index S of the resolution, it is calculated by di / fi using the distance di between the three-dimensional coordinates corresponding to the pixel of interest and the three-dimensional position of the viewpoint of interest and the focal length fi of the viewpoint of interest. Reference numeral i indicates a symbol for identifying the viewpoint of interest at a plurality of imaging viewpoints. The method for calculating the resolution index S is not limited to the above, and other methods for determining the size of the subject appearing in the image of the viewpoint of interest may be used. For example, the angle of view may be used instead of the focal length.

The higher the value of the resolution index S, the higher the resolution of the three-dimensional coordinates corresponding to the pixel of interest. For the texture distortion index W, the direction vector CA of the viewpoint of interest is calculated from the position coordinates of the viewpoint of interest in the three-dimensional space and the three-dimensional coordinates corresponding to the pixels of interest, and the direction vector CA and the normal vector calculated in the preparation are performed. As a result of the inner product with and, cos θ is used. However, the index of texture distortion is not limited to this, and may be set by using an angle, a weight based on the angle, or the like, as in the content described in the method of calculating the evaluation value Q.

The larger the value of the texture distortion index W, the more the three-dimensional coordinates corresponding to the pixel of interest are captured from a position where the viewpoint of interest is closer to the front. Therefore, the larger the value of the evaluation value V calculated by the product of the above indexes, the more the three-dimensional coordinates corresponding to the pixel of interest are imaged from the viewpoint of interest with high resolution and less distortion of the texture. In the present embodiment, the evaluation value V using each index at the same ratio is set, but the present invention is not limited to this. If you want to emphasize the resolution, you may make adjustments such as adding a power term to the resolution index and then multiplying it with the texture distortion index. The second calculation unit 707 uses the calculated evaluation value V at all viewpoints as an index of the distortion of the texture used for calculating the evaluation value V and the pixel coordinates (UV coordinates) for each imaging viewpoint held together with the image generation unit. Output to 708.

Returning to FIG. 8, in step S809, the image generation unit 708 determines the pixel value of the pixel of interest based on the evaluation value V of all the imaging viewpoints of the pixel of interest acquired from the second calculation unit 707. The image generation unit 708 determines the pixel value based on a plurality of captured images in consideration of the deterioration of image quality due to the deviation of the texture between the polygons. The method of determining the pixel value will be described below.

(Method of determining pixel value)
The method of determining the pixel value in S809 will be described in detail. The image generation unit 708 determines the pixel value based on a plurality of captured images. First, the imaging viewpoint j1 having the maximum evaluation value V is detected from the evaluation values V of all viewpoints, and the evaluation value Vj1 is acquired. Next, from the imaging viewpoints other than the imaging viewpoint j1, the imaging viewpoint j2 having the maximum evaluation value V (that is, the imaging viewpoint j2 having the second largest evaluation value V when the imaging viewpoint j1 is included) is detected. The evaluation value Vj2 of the imaging viewpoint is acquired. Further, from the detected evaluation values Vj1 and Vj2 of the imaging viewpoints j1 and j2, the blend ratio (mixing ratio) B of the captured images of the imaging viewpoints j1 and j2 is determined based on the equations 2 and 3. The blend ratio of other viewpoints is set to 0.

Bj1 = Vj1 / (Vj1 + Vj2) ... Equation 2
Bj2 = Vj2 / (Vj1 + Vj2) ... Equation 3
In the present embodiment, the imaging viewpoint is fixed, and the image is taken so as to surround the subject as shown in FIG. Therefore, it is highly possible that the polygon of interest and the polygon adjacent thereto are both imaged from the same imaging viewpoint and from adjacent imaging viewpoints. Further, when the subject is a person or the like, the adjacent polygons in the 3D polygon data of the subject are smoothly connected and changed. Since the evaluation value V is set using the index of the resolution and the texture distortion based on the degree of front of the polygon and the imaging viewpoint, the imaging viewpoints having higher evaluation values may be similar between adjacent polygons. high. Therefore, by giving a high blend ratio to a plurality of imaging viewpoints having higher evaluation values and blending them, it is possible to suppress abrupt texture switching between polygons. In the present embodiment, the blend ratio is determined using only the top two viewpoints, but the present invention is not limited to this, and an arbitrary number of viewpoints may be selected to determine the blend ratio. If the evaluation value V is an error value, that is, a negative value in all imaging viewpoints, the pixel value of the pixel of interest may be determined by a predetermined pixel value without setting the blend ratio.

Finally, the pixel value of the pixel of interest is determined by the weighted sum based on the determined blend ratio. The pixel value of each imaging viewpoint is acquired according to the pixel coordinates acquired from the second calculation unit 707. However, the acquisition of the pixel value of each imaging viewpoint is not limited to this, and the peripheral pixel values such as the average pixel value of the peripheral pixels may be considered around the acquired pixel coordinates.

Returning to FIG. 8, in step S810, the image generation unit 708 determines whether or not the pixel values have been determined for all the pixels of the texture image. If the result of the determination in step S810 is false (No), the process returns to step S807. On the other hand, when the result of the determination in step S810 is true (Yes), the generated texture image is output to the secondary storage device 104, the external storage device 108, or the display device 109, and the series of processes is completed. The above is the process of generating the textured 3D polygon model executed by the image processing apparatus 100 in the present embodiment.

As described above, in the present embodiment, the generation of the texture image is roughly divided into two steps, the first step ensures the resolution on the texture image, and the second step captures a plurality of imaging viewpoints for each pixel. The image is blended to determine the pixel value. As a result, it is possible to generate a texture image having high resolution and inconspicuous texture steps between polygons, and to generate a high-quality textured 3D polygon model.

The 3D polygon model generated in this embodiment is used to generate a virtual viewpoint image. For example, when an instruction to generate a virtual viewpoint image is given by using a virtual viewpoint image generation application of a portable terminal such as a tablet or a smartphone, a 3D polygon model is transmitted to the portable terminal. Then, in the portable terminal, a virtual viewpoint image may be generated based on the viewpoint specified by the user and the 3D polygon model. In addition, the server does not send the 3D polygon model to the portable terminal, the server acquires the information of the viewpoint specified by the portable terminal, and the server generates a virtual viewpoint image based on the acquired viewpoint information and the 3D polygon model. You may try to do it. Then, the generated virtual viewpoint image may be transmitted to the portable terminal.

<Embodiment 2>
In the first embodiment, when determining the pixel value of the texture image, the pixel value of the captured image of the higher imaging viewpoint having a high evaluation value V is blended based on the ratio based on the evaluation value V. As a result, a high-quality textured 3D polygon model with no texture step between polygons is generated. In the second embodiment, the evaluation value V based on the resolution index and the texture distortion index is used as a reference when determining the blend ratio, and the texture distortion index is used as the rendering ratio when determining the pixel value. The form used for is described. The outline and significance of this embodiment will be described below.

In the present embodiment, first, the priority for selecting the imaging viewpoint to be blended is determined by the evaluation value V calculated by the product of the resolution index S determined for each imaging viewpoint and the texture distortion index W. In both the resolution index S and the texture distortion index W, the higher the value, the higher the possibility that a high-quality texture image can be generated. Therefore, the higher the priority of the imaging viewpoint, the higher the possibility that a high-quality texture image can be generated. Next, for each imaging viewpoint, an imaging viewpoint that holds a priority higher than its own priority is detected. Then, at least one imaging viewpoint having a large texture distortion index W among the detected imaging viewpoints is selected as the imaging device used for blending. Also, the corresponding priority is determined as the blend weight. Then, the blend weight and the selected imaging viewpoint in all the imaging viewpoints are integrated to calculate the rendering weight for determining the pixel value for each imaging viewpoint.

For example, in the arrangement configuration of the image pickup apparatus shown in FIG. 11, the image pickup viewpoints 1102 and 1103 are arranged on the left and right sides of the polygon 1101 in the subject 1100. Here, it is assumed that the resolution index is about the same for polygons at both of the imaging viewpoints 1102 and 1103, and the texture distortion index is about the same although there is a difference in the left-right direction. Therefore, the evaluation value, which is the product of the indexes, is about the same in both imaging viewpoints. In this case, according to the first embodiment, the pixel value is determined as follows. That is, the pixels included in the texture region 1105 corresponding to the polygon 1101 on the texture image 1104 blend the pixel values in the captured image of the imaging viewpoint 1102 and the pixel values in the captured image of the imaging viewpoint 1103 at the same ratio. The pixel value is determined. By blending and determining the pixel value, it is possible to generate a high-quality texture with no texture step between polygons or inside the polygon. However, as described above, even if the image captures the same portion, the texture projected on the polygon may be displaced or distorted between different imaging viewpoints due to the influence of the error included in the 3D polygon data. Due to this effect, the texture region 1105 determined by blending has a deteriorated resolution as compared with the case where the imaging viewpoints 1102 and 113 are used alone. Therefore, in the present embodiment, by optimally blending the imaging viewpoints and suppressing unnecessary blending, a texture image having no texture step and a higher resolution is generated.

In the present embodiment, the final rendering weight in the generation of the virtual viewpoint image is determined by utilizing the smooth change of the three-dimensional coordinates and the normal direction of the 3D polygon data in a subject such as a person. After detecting all imaging viewpoints whose priority is equal to or higher than the evaluation value V of the texture distortion index W and the resolution index S, the blending of each imaging viewpoint is based on the texture distortion index W. Determine the weight to use for. A specific description will be given with reference to FIG. FIG. 11B shows a bird's-eye view of FIG. 11A. The surface 1106 corresponds to the polygon 1101, and the arrows attached to the surface 1106 indicate the three-dimensional coordinates and the normal direction for each pixel included in the texture area 1105. In the pixels existing on the surface 1106, the imaging viewpoints 1102 and 1103 having the same evaluation value, that is, the same priority are detected. In the detected imaging viewpoint, a blend weight is set as the index W of the texture distortion is larger than the index W of the texture distortion of the imaging viewpoint with respect to each normal existing on the surface 1106. Then, based on the set weight, the captured images of the imaging viewpoint 1102 and the imaging viewpoint 1103 are blended to determine the pixel values of the pixels included in the texture region 1105.

The normal of the surface 1106 smoothly changes from a direction close to parallel to the direction vector connecting the subject and the imaging viewpoint 1102 to a direction close to parallel to the direction vector connecting the subject and the imaging viewpoint 1103 as it advances to the right. .. Therefore, the blend ratio set using the texture distortion index W calculated by the degree of front of the imaging viewpoint with respect to the normal changes smoothly so that the blend ratio of the imaging viewpoint 1103 increases from the imaging viewpoint 1102 to the right. Can be made to. As a result, it is possible to suppress unnecessary blending and generate a high-quality textured 3D polygon model with high resolution and little texture deviation.

Hereinafter, the processing performed by the image processing apparatus 900 of the present embodiment will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram showing a functional configuration of the image processing device 900 in the present embodiment. FIG. 10 shows a flowchart of an image processing method performed by the image processing apparatus 900 in the present embodiment. In the present embodiment, the same configuration and the same processing as in the first embodiment are designated by the same reference numerals as those in the first embodiment, and the description thereof will be omitted. Further, since the block diagram showing the hardware configuration of the image processing device 900 in the present embodiment is the same as that in FIG. 1, the description thereof will be omitted.

As shown in FIG. 9, the image processing apparatus 900 includes an image data acquisition unit 701, a viewpoint information acquisition unit 702, a 3D polygon acquisition unit 703, a UV coordinate acquisition unit 704, a first calculation unit 705, and an area, as in FIG. It has a determination unit 706. Further, the image processing apparatus 900 has a second calculation unit 901, a blend ratio determination unit 902, and an image generation unit 903. The image processing device 900 functions as each component shown in FIG. 9 by executing the program stored in the ROM 103 by the CPU 101 using the RAM 102 as the work memory, and executes a series of processes shown in the flowchart of FIG. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and the image processing device 100 is configured so that a part or all of the processes is performed by one or a plurality of processing circuits other than the CPU 101. May be good. Hereinafter, the flow of processing performed by each component will be described, but since steps S1001 to S1006 are the same as steps S801 to S806 in FIG. 8 of the first embodiment, the description thereof will be omitted.

In step S1007, the second calculation unit 901 sets the pixel of interest for determining the pixel value in the pixels included in the texture image.

In step S1008, the second calculation unit 901 calculates the evaluation value V of the pixel of interest for each imaging viewpoint using the texture image acquired from the region determination unit 706. The evaluation value V calculated here is used as an index when determining the pixel value of the texture in the pixel of interest. Since the method of calculating the evaluation value V is the same as that calculated in the process of step S808 of the first embodiment, the description thereof will be omitted. The second calculation unit 901 outputs the calculated evaluation value V at all viewpoints to the blend ratio determination unit 902 together with the index of the distortion of the texture used for calculating the evaluation value. Further, the second calculation unit 901 outputs the pixel coordinates for each imaging viewpoint held to the image generation unit 903.

In step S1009, the blend ratio determination unit 902 determines the priority of the imaging viewpoint for each imaging viewpoint from the evaluation value V of each imaging viewpoint acquired from the second calculation unit 901. In the present embodiment, the evaluation value V itself is determined as the priority. However, the method of expressing the priority is not limited to the above, and an adjustment such as taking the product of the evaluation value V and a value smaller than 1 may be made at the imaging viewpoint for which the priority is to be lowered in advance.

In step S1010, the blend ratio determination unit 902 blends each imaging viewpoint for each priority by using the priority for each imaging viewpoint determined in step S1009 and the texture distortion index acquired from the second calculation unit 901. Determine the ratio. This will be described in detail below.

First, the lowest priority among all the priorities is selected as the priority to be focused on. After that, with respect to the priority of interest (hereinafter referred to as the priority of interest), each time the following processing is completed, the priority that has not yet been selected as the priority of interest is set as the priority of interest. It is selected and the same process is repeated for all priorities until the process is completed.

Next, the imaging viewpoint that holds the priority of interest or higher is detected among all the imaging viewpoints, and the imaging viewpoint used for blending is selected from the detected imaging viewpoints based on the index of texture distortion. In the present embodiment, an imaging viewpoint having a texture distortion index within the top two is selected from the imaging viewpoints. However, it is not always necessary to select two imaging viewpoints, and one imaging viewpoint having the maximum texture distortion index among all the imaging viewpoints may be selected.

Further, the blend weight is determined for each imaging viewpoint with respect to the imaging viewpoint selected by the priority of attention. In the present embodiment, the blend weight is determined so that the weight decreases linearly as the index of texture distortion decreases according to the index of texture distortion of the selected imaging viewpoint. Specifically, when the total value Wsum of the indexes indicating the texture distortion of the selected imaging viewpoint and the index values W1 and W2 of the respective imaging viewpoints, the blend weights are B1 = W1 / Wsum and W2 = W2 / Wsum. To do. However, the method for calculating the blend weight is not limited to this, and various methods such as non-linear weight reduction and abrupt weight reduction from a certain angle or more can be used.

The blend ratio determination unit 902 determines the viewpoint number of the imaging viewpoint selected in the focus priority and the blend weight for each of the imaging viewpoints as blend information for the focus priority. Then, the same processing as described above is performed for all the priorities, and the blend ratio determination unit 902 outputs the blend information for each priority to the image generation unit 903.

In step S1004, the image generation unit 903 integrates the blend information for each priority acquired from the blend ratio determination unit 902, and determines the rendering weight for each imaging viewpoint. Then, the pixel value of the pixel of interest is determined according to the determined rendering weight.

In the present embodiment, first, the rendering weights of all imaging viewpoints are initialized to 0. After that, the blend information at all the priorities is confirmed for each imaging viewpoint, and the priority including the number of the own imaging viewpoint is detected. Then, for each detected priority, the product of the priority and the blend weight of its own imaging viewpoint is obtained. Further, in all the detected priorities, the sum of the products of the priorities obtained for each priority and the blend weights is obtained, and the result is used as a provisional rendering weight at the imaging viewpoint. Finally, after determining the temporary rendering weights for all imaging viewpoints, the result of dividing the rendering weights of each imaging viewpoint by the sum of the temporary rendering weights of all imaging viewpoints is determined as the rendering weights of each imaging viewpoint. ..

The image generation unit 903 determines the pixel value of the pixel of interest by the weighting sum based on the determined rendering weight. The pixel value of each imaging viewpoint is acquired according to the pixel coordinates acquired from the second calculation unit 901. However, the acquisition of the pixel value of each imaging viewpoint is not limited to this, and the peripheral pixel values such as the average pixel value of the peripheral pixels may be taken into consideration around the acquired pixel coordinates.

The above is the processing performed by the image processing apparatus 900 of the present embodiment. According to the configuration described above, it is possible to suppress unnecessary blending and generate a high-quality textured 3D polygon model with high resolution and reduced texture deviation.

<Other Embodiments>
The embodiment of the present invention is not limited to the above embodiment, and various embodiments can be taken. For example, in the above embodiment, two indexes, a resolution index and a texture distortion index, are used as the method for calculating the evaluation value V, but the present invention is not limited to this, and a plurality of indexes may be additionally used. For example, when detecting the periphery of the contour of the subject on the image of the imaging viewpoint and determining the pixel value of the texture image, an index may be added so that the weight of the pixel around the detected contour becomes small. As a result, when the 3D polygon data is slightly larger than the actual subject due to the error included in the 3D polygon data, deterioration of image quality due to erroneous use of pixel values other than the subject can be suppressed.

Further, in the second embodiment, when determining the blend ratio, it is not distinguished whether or not each imaging viewpoint is located in the clockwise direction or the counterclockwise direction with respect to the normal of the subject. Not limited to. When selecting two imaging viewpoints at each priority, a method such as selecting an imaging viewpoint having a high index of texture distortion from each direction may be used. The same applies to the determination of the blend ratio in the first embodiment.

Further, in the above embodiment, when securing the texture area of polygons in the texture image, processing is performed for each polygon included in the 3D polygon data, but the present invention is not limited to this. A plurality of adjacent polygons may be collectively treated as one large polygon to secure a texture area. Further, in the above embodiment, when determining the pixel value of each pixel in the texture image, processing is performed for each pixel, but the present invention is not limited to this. A plurality of adjacent pixels may be processed together.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image processing device 708 Image generator

Claims (29)

An image processing device that generates a texture image corresponding to three-dimensional polygon data of an object based on a plurality of captured images based on capturing an object using a plurality of imaging devices.
Each pixel value of the plurality of pixels or the plurality of pixel groups constituting the region corresponding to the polygon constituting the three-dimensional polygon data is set to one or more captured images of the plurality of captured images for each pixel or pixel group. And the decision-making means to decide based on
An image processing apparatus comprising: a generation means for generating the texture image based on a pixel value determined by the determination means. The image processing apparatus according to claim 1, wherein the texture image includes at least one of color information, luminance information, and saturation information. The determination means according to claim 1 or 2, wherein the image captured image used for determining the pixel value is determined for each pixel or pixel group based on the respective focal length or angle of view of the image pickup apparatus. The image processing apparatus described. The determination means determines an image to be used for determining the pixel value for each pixel or pixel group based on the position in the object corresponding to the pixel or the pixel group constituting the region and the position of the image pickup device. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is used. The determination means obtains a pixel or a pixel of an image captured image used for determining the pixel value based on a distance between a position in the object corresponding to a pixel or a group of pixels constituting the region and a position of the image pickup device. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is determined for each group. The determination means determines the pixel value of the pixel or pixel group constituting the region based on the direction vector connecting the position in the object corresponding to the pixel or pixel group constituting the region and the position of the image pickup apparatus. The image processing apparatus according to any one of claims 1 to 5, wherein the captured image used in the above is determined for each pixel or pixel group. The determination means according to any one of claims 1 to 6, wherein the determination means determines the pixel value based on two or more captured images out of the plurality of captured images for each pixel or pixel group. The image processing apparatus described. It further has a first calculation means for calculating the first evaluation value of each of the plurality of captured images for each of the pixels or pixel groups.
The claim means that the determination means determines two or more captured images used for determining the pixel value for each pixel or pixel group based on the first evaluation value calculated by the first calculation means. Item 7. The image processing apparatus according to item 7. The determination means determines the mixing ratio of the two or more captured images used for determining the pixel value based on the first evaluation value, and determines the pixel value based on the determined mixing ratio. 8. The image processing apparatus according to claim 8. The determination means is characterized in that the mixing ratio of the two or more captured images is determined so that the mixing ratio of the captured images having a large first evaluation value is large among the two or more captured images. Item 9. The image processing apparatus according to item 9. Claims 8 to 10 are characterized in that the determination means determines two or more captured images having a first evaluation value larger than other captured images as two or more captured images used for determining the pixel value. The image processing apparatus according to any one of the above items. 11. The determination means according to claim 11, wherein the determination means determines two or more captured images including the captured image having the maximum first evaluation value as two or more captured images used for determining the pixel value. Image processing equipment. The determination means determines two or more captured images including the captured image having the maximum first evaluation value and the captured image having the second largest evaluation value as two or more captured images used for determining the pixel value. The image processing apparatus according to claim 12. The determination means determines the pixel value of the pixel or the pixel group based on the priority for determining the pixel value set for each of the plurality of image pickup devices for each pixel or the pixel group. The image processing apparatus according to any one of claims 1 to 13, wherein the captured image to be used is determined. The determination means is characterized in that the mixing ratio of the two or more captured images used for determining the pixel value is determined based on the priority, and the pixel value is determined based on the determined mixing ratio. The image processing apparatus according to claim 14. The determination means is further used to determine the pixel value of the pixel or the pixel group based on the direction vector connecting the position in the object corresponding to the pixel or the pixel group constituting the region and the position of the image pickup device. The image processing apparatus according to claim 14 or 15, wherein the captured image is determined. The determination means is
In each of the two or more captured images used for determining the pixel value, the weight used for determining the mixing ratio is determined for each of the priorities.
The image processing apparatus according to claim 15, wherein the mixing ratio of each of the two or more captured images is determined based on the weight determined for each priority. The weights are a direction vector connecting a position in the object corresponding to the pixel or a group of pixels constituting the region and a position of the image pickup apparatus, and a position in the object corresponding to the pixel or the group of pixels constituting the region. The image processing apparatus according to claim 17, wherein the image processing apparatus becomes larger as the normal direction becomes closer to parallel. The image processing apparatus according to any one of claims 1 to 18, further comprising an associating means for associating the polygon with the region based on one of the plurality of captured images. .. The associating means determines the one captured image based on the resolution of the portion corresponding to the polygon included in the captured image, and based on the determined one captured image, the polygon and the region The image processing apparatus according to claim 19, wherein the image processing apparatus is associated with. The image processing apparatus according to claim 20, wherein the resolution is represented by the number of pixels constituting a portion corresponding to the polygon included in the captured image. The associating means is based on the angle formed by the direction vector connecting the position in the object corresponding to the pixel or the pixel group constituting the region and the position of the image pickup apparatus and the normal direction of the polygon. The image processing apparatus according to any one of claims 19 to 21, wherein one captured image is determined, and the polygon and the region are associated with each other based on the determined captured image. The image processing apparatus according to claim 22, wherein the associating means determines the one captured image based on the inner product of the direction vector and the normal direction of the polygon. The associating means determines the one captured image based on the second evaluation value based on the resolution of the region corresponding to the polygon included in the captured image, and based on the determined one captured image, The image processing apparatus according to any one of claims 19 to 23, wherein the polygon is associated with the region. The second evaluation value is further based on the angle formed by the direction vector connecting the position in the object corresponding to the pixel or the pixel group constituting the region and the position of the image pickup apparatus and the normal direction of the polygon. The image processing apparatus according to claim 24. It further has a second calculation means for calculating the second evaluation value of each of the plurality of captured images.
The claim is characterized in that the associating means determines an image captured in which the second evaluation value calculated by the second calculation means is larger than the evaluation value of another image, and determines the one image. 24 or 25. The image processing apparatus. The image processing apparatus according to claim 26, wherein the associating means determines the captured image having the maximum second evaluation value as the one captured image. An image processing method for generating a texture image corresponding to three-dimensional polygon data of an object based on a plurality of captured images acquired by imaging the object using a plurality of imaging devices.
Each pixel value of the plurality of pixels or the plurality of pixel groups constituting the region corresponding to the polygon constituting the three-dimensional polygon data is set to one or more captured images of the plurality of captured images for each pixel or pixel group. And the decision process to decide based on
An image processing method comprising a generation step of generating the texture image based on a pixel value determined by the determination step. A program for causing a computer to function as the image processing device according to any one of claims 1 to 27.

Images