JP5997664B2 - Generating texture image data - Google Patents

Description

  The present invention relates to a method for generating texture image data to be applied to a polygon of a three-dimensional model.

  In a three-dimensional map that three-dimensionally represents the shape of a feature, it is required to generate a three-dimensional model of the feature with high accuracy and a low processing load. In the three-dimensional model, in order to improve the reality, a method is used in which a texture generated based on an image obtained by photographing each object is pasted on the surface of a polygon. For example, Patent Documents 1 and 2 each disclose a technique in which a part of a photograph of a feature photographed is cut out and used as a texture.

JP 2003-6680 A JP 2004-102474 A

  However, there may be various other features around the feature to be generated by the 3D model, so the captured feature does not necessarily show the entire feature. Not exclusively. In addition, an image showing the entire feature is not always high in image quality in all parts. In order to avoid such problems, even if you try to generate one texture image by cutting out each good part from multiple images and synthesizing like patchwork, unnaturalness remains in the joint part of the patch, On the contrary, the texture may become uncomfortable. In view of these problems, an object of the present invention is to enable generation of a texture with improved reality based on a captured image.

The present invention
A texture image data generation method for generating image data of a texture image to be applied to a polygon constituting a three-dimensional model of a feature by a computer,
As the steps executed by the computer,
(A) reading a plurality of image data obtained by photographing a surface corresponding to the polygon of the feature;
(B) For each pixel constituting the texture image, obtaining color information of a location corresponding to the pixel from the plurality of image data;
(C) It can comprise as a generation method of texture image data provided with the step of generating the texture image by determining the color of the pixel based on the acquired color information.

  According to the present invention, color information of a portion corresponding to a pixel to be processed is acquired from a plurality of image data, and based on this, the color of the pixel is determined. As described above, since a plurality of pieces of image data are used for generating the texture, it is possible to compensate for a defect existing in each piece of image data. In addition, since texture images are synthesized in units of pixels, it is possible to generate natural texture image data with no seams or the like as compared with a method of synthesis like patchwork.

The plurality of image data used in the present invention may be those obtained by photographing the surface to be processed (hereinafter sometimes referred to as “target surface”), and may be those photographed at the same point and photographing angle. The images may be taken from various points and shooting angles. Moreover, the whole symmetry plane may be reflected, or a part of the symmetry plane may be reflected.
Various methods can be used for determining the color of a target pixel from color information acquired from a plurality of image data. For example, an average value, a median value, and a color having a maximum frequency distribution among a plurality of color information can be used.

In the texture image data generation method of the present invention,
The step (B)
(B1) obtaining a correspondence relationship between a plurality of feature points in the plurality of image data;
(B2) The method may include a step of specifying a location corresponding to the pixel on the basis of a plurality of feature points located around the pixel.

In this way, the location corresponding to the pixel to be processed can be specified with high accuracy in each image, and the image quality of the generated texture image can be improved.
One or two feature points may be used for specifying a location corresponding to a pixel to be processed, but three or more feature points may be used to specify a two-dimensional position with high accuracy. preferable.
For the location corresponding to the pixel to be processed, the relative position with respect to the pixel to be processed is obtained from the feature point, and the location at the same relative position with respect to the feature point in each image may be obtained.
Feature points include, for example, the corners of symmetry planes, windows on the surface of features, handrails, vertices of signboards and other structures, and points where the color or shape of the object changes abruptly. Can do. Further, an image feature quantity related technique represented by SIFT (Scale-Invariant Feature Transform) may be used. If such a technique is used, since a large number of feature points can be associated between a plurality of images based on the color of the image, it is possible to improve the accuracy of specifying a location corresponding to a pixel.

In the texture image data generation method of the present invention,
The step (C)
It is good also as what determines the color of the said pixel except the part shielded by the other feature among the said image data.
Since the color information of the shielded part is different from the color of the target surface, the color determination accuracy can be improved by excluding it in this way.

Thus, in an aspect excluding the shielded part,
The step (A) further reads a photographing condition of the image data,
The step (C) refers to a three-dimensional model representing the feature and a three-dimensional shape around the feature, and identifies the shielded portion based on the three-dimensional model and the imaging condition. It may be a thing.
The shooting condition refers to information necessary for specifying the state of a shot image, such as a shooting position, a shooting direction or angle, and a field angle at the time of shooting.
If virtual “shooting” is performed based on shooting conditions in a virtual space consisting of a three-dimensional model, it is relatively easy to detect the part of the target surface that is covered by other features in the obtained image data. Can be specified. By applying the region thus identified, it is possible to relatively easily and accurately identify the shielded portion in each image used for generating the texture.
In addition to such a method, the shielded portion may be specified by analyzing, for example, discontinuity in color or shape in the image data. When using image feature values, not only can a large number of feature points be obtained between multiple images, but also a normal vector can be obtained for each feature point. Based on the continuity, it is also possible to identify the shielded part.

In the texture image data generation method of the present invention,
The step (A) may convert the surface corresponding to the polygon into an image viewed from the normal direction of the surface, and read the plurality of captured image data.
Such image conversion can be performed by, for example, specifying a direction from which the target surface was imaged based on the imaging conditions of each image and converting the image to a state viewed from the normal direction based on this. . As another method, affine transformation may be performed so that the shape of the target surface of each image matches one of the reference shapes. The conversion of the image can be performed by various other methods.
As described above, if the processing is performed after converting the image data into a unified shape, the location corresponding to the pixel to be processed can be easily specified, the processing load can be reduced, and the accuracy can be improved. You can also.

  The present invention does not necessarily include all the various features described above, and may be configured by omitting or combining some of them as appropriate. Moreover, you may comprise this invention as a texture image data generation apparatus other than the aspect as the above-mentioned texture image data generation method. Furthermore, it may be configured as a computer program that causes a computer to generate texture image data, or may be configured as a computer-readable recording medium that records the computer program.

It is explanatory drawing which shows the structure of a three-dimensional model production | generation system. It is a flowchart of a three-dimensional point cloud production | generation process. It is a flowchart of a three-dimensional model generation process. It is a flowchart (1) of a texture generation process. It is a flowchart (2) of a texture production | generation process. It is a flowchart (3) of a texture production | generation process.

A. System configuration:
FIG. 1 is an explanatory diagram showing a configuration of a three-dimensional model generation system. The three-dimensional model generation system is a system that takes a photograph of an area to be processed from an aircraft and generates a three-dimensional model representing a three-dimensional shape of each feature by analyzing the photograph. A device that is mounted on an aircraft and acquires image data of a local photograph is referred to as a photographing management device 100. A device that generates a three-dimensional model from image data is referred to as a three-dimensional model generation device 200. The texture image data generation device of the present invention is realized as one function of the three-dimensional model generation device 200. Hereinafter, each structure is demonstrated in order.

(1) Configuration of the imaging management device 100:
The imaging management device 100 is mounted on an aircraft. The aircraft may be either fixed wing or rotary wing, and may be manned / unmanned.
The photographing management apparatus 100 includes a photographing camera 101, a GPS (Global
Positioning System) 102 and functional blocks shown in the figure. The functional block can be configured by installing a computer program for realizing each function in the computer, or may be configured by hardware.
The camera 101 is for taking a photograph used for analysis for generating a three-dimensional model. In order to improve the analysis accuracy, a high-resolution photograph is desirable. In the present embodiment, a still image digital camera capable of shooting at a high resolution is used. If a sufficient resolution can be obtained, a video camera may be used.

The shooting sequence storage unit 112 stores flight paths and shooting points of aircraft for shooting. The shooting point is set in consideration of the angle of view of the camera 101, and is set so that 20 or more photographs are sequentially taken in the state to be processed with about 60% or more overlapping. Has been. A method for setting the flight path will be described later. The flight route or the like may be set in advance by another computer or the like, and may be stored in the imaging sequence storage unit 112 or may be set in the imaging sequence storage unit 112.
When using a manned aircraft, it is the pilot's duty to fly the aircraft according to the flight path. In such a case, the imaging sequence storage unit 112 may appropriately provide information for supporting the flight along the set flight path to the pilot.
When an unmanned aircraft is used, the imaging sequence storage unit 112 may be configured to transmit information related to the flight path to the control device that controls the flight.

The shooting control unit 110 controls the camera 101 and takes a picture of the processing target area at the shooting point set in the shooting sequence storage unit 112. Since there is an error between the actual flight path of the aircraft and the path stored in the imaging sequence storage unit 112, the imaging control unit 110 determines that the vicinity of the preset imaging point has been reached based on the error. You may make it image | photograph at the time of this.
Further, the shooting sequence is not necessarily limited to the method of specifying the shooting point, but may be set so that shooting is performed at a constant time period. In such a case, the shooting control unit 110 controls the camera 101 to perform shooting at set time intervals.
The shooting control unit 110 acquires position information of a shooting position from the GPS 102 in synchronization with shooting by the camera 101 and records the information in the image recording unit 120 in association with the shooting by any method.
The imaging control unit 110 may be configured to automatically perform imaging, and may notify an imaging person on board an imaging timing.

(2) Configuration of the three-dimensional model generation apparatus 200:
The three-dimensional model generation apparatus 200 includes the functional blocks shown in the figure. These functional blocks are configured by installing a computer program for realizing each function in a computer, or may be configured in hardware.
Hereinafter, the function of each functional block will be described.

The image data storage unit 201 reads and stores image data shot and collected by the shooting management apparatus 100 and position information of shooting positions. Transfer of image data from the imaging management apparatus 100 to the image data storage unit 201 can be performed via a wired, wireless, or recording medium.
The three-dimensional point group generation unit 202 generates a three-dimensional point group using the image data and the shooting position of the image data storage unit 201. A three-dimensional point group is a number of points to which three-dimensional coordinate values and normal vectors are assigned. The normal vector means a normal vector of a surface where each point is estimated to exist.
In this embodiment, when the image data is arranged in time series, the preceding and following image data are taken with an area of about 60% or more overlapping. That is, the building in the target area is photographed by a plurality of image data having different photographing points. Therefore, if points corresponding to each other in a plurality of image data can be specified, the three-dimensional coordinates of the points can be specified by the same principle as that of so-called triangulation. Various techniques are known for identifying corresponding points in a plurality of image data. In this embodiment, a known technique called SIFT (Scale-Invariant Feature Transform) is used as an example of an image feature amount. Applied.
The three-dimensional point group data storage unit 203 stores the three-dimensional point group obtained by the three-dimensional point group generation unit 202. Since it is a point cloud obtained by analyzing a portion shown in the image data, the three-dimensional point cloud mainly represents an outer wall portion such as a building and an outer surface portion such as a roof. Also, there are many buildings other than buildings such as the ground, roads, roadside trees, vehicles, etc. in the image, and as long as the correspondence in the image is specified, a 3D point cloud can be generated. The point cloud includes many things that represent other than these buildings. Each three-dimensional point group has a three-dimensional coordinate value and a normal vector, but these coordinate value and normal vector are not defined in the earth coordinate system and are set for analysis. It is a coordinate value in a fictitious three-dimensional space.
In this way, 3D polygons with position coordinates are generated with accuracy that can be used as a map for each building from a 3D point group that is not specified in the earth coordinate system and is not divided into buildings. It is the function of each functional block shown below.

The gravity direction analysis unit 204 analyzes the gravity direction of the point group. Usually, when the geographical position is represented in the Cartesian coordinate system, the direction of gravity coincides with the Z-axis direction, but the coordinate system of the three-dimensional point group is not defined in the Earth coordinate system. The Z axis is not always in the direction of gravity. The gravitational direction analysis unit 204 functions as a first stage for associating the coordinate system of the point group with the earth coordinate system by obtaining the gravitational direction by analysis.
The point cloud separation processing unit 205 separates the point cloud into building units. The three-dimensional point cloud is obtained by obtaining the three-dimensional coordinates of the points shown in the image, and is not separated from the building ground or the like unless it is separated in units of buildings. The point group separation processing unit 205 has a function of analyzing the three-dimensional point group and separating it into point groups for each building.
The building frame generation unit 206 generates a building frame representing a two-dimensional outer shape based on the point group separated for each building.
The alignment processing unit 207 has a function of associating a coordinate system other than the gravity direction with the earth coordinate system by fitting a plurality of building frames to the two-dimensional map stored in the two-dimensional map data 210. The alignment processing unit 207 does not adapt the point group separated for each building to the two-dimensional map one by one. The whole of a plurality of buildings represented by point clouds is fitted to a two-dimensional map as a whole while maintaining the relative relationship in general. In this way, alignment can be performed with high accuracy by adapting the entire plurality of buildings.
The polygon generation unit 208 generates side and top polygons based on the point group for each building whose two-dimensional shape is determined by the above processing.
The texture generation unit 209 generates a texture to be pasted on the surface of the generated polygon based on the image data stored in the image data storage unit 201.
The three-dimensional polygon and texture generated by the above processing are stored in the three-dimensional model data 211.

B. Three-dimensional point cloud generation processing:
When the flight path and shooting point are determined, shooting is performed while flying around the target. The captured image data is sequentially recorded in the image recording unit 120 of the imaging management apparatus 100 together with the position coordinates of the imaging point as described with reference to FIG. When the photographing is completed, the image data and the position coordinates are input to the three-dimensional model generation apparatus 200, and a three-dimensional point group is generated. The image data can be read into the 3D model generation apparatus 200 by various methods. In this embodiment, the image data is read via a recording medium.

FIG. 2 is a flowchart of the three-dimensional point group generation process. This is a process executed by the 3D point group generation unit 202 of the 3D model generation apparatus 200, and is a process executed by the CPU of the 3D model generation apparatus 200 in terms of hardware.
When the process is started, the three-dimensional model generation apparatus 200 reads image data and position information (step S10). Then, the feature points between the images are associated by matching with the image feature amount (step S11). The state of the feature point association is shown in the figure. As shown in the figure, feature points (four crosses in the figure) are extracted from each of image 1 and image 2, and the correspondence between the feature points is recognized. Matching based on image feature amounts is an analysis technique for extracting feature points based on color components in an image and obtaining a correspondence relationship between images. Since the analysis principle and algorithm are well known, detailed description thereof is omitted. In step S11, various methods such as SIFT can be used as long as the feature points between the images are associated with each other.

When the correspondence between the feature points is obtained in this way, the three-dimensional coordinates and normal vectors of each feature point can be calculated (step S12). The calculation principle is shown in the figure. Each image used in the embodiment is provided with shooting point position information, so two or more shooting points 1 and 2 correspond to feature points for which a correspondence relationship is obtained between a plurality of images. Will be attached. If the position of the feature point in the image taken based on the angle of view from the shooting points 1 and 2 is analyzed, the direction in which the feature point was viewed from each of the shooting points 1 and 2 can be determined. It is possible to specify the relative positional relationship of the feature points with respect to the coordinate values of 1 and 2. The position coordinates of the feature points obtained here are not absolute position coordinates in the earth coordinate system, but are coordinate values on fictitious three-dimensional coordinates (xa, ya, za) set in the space for analysis. is there.
The normal vector is a vector in the normal direction of the estimated surface where each feature point is estimated to exist. In the embodiment, on the premise that a large number of feature points are extracted from corresponding images and the relative positional relationship between the feature points in each image is maintained, the correspondence relationship of the feature points between the images is recognized. . In this way, the estimated surface where the feature point exists is specified from the relative positional relationship between the feature points, and the normal vector can be obtained.
The relative positional relationship between these images, the calculation of three-dimensional coordinates and normal vectors are also calculated using SfM (Structure
(from Motion) technique and a technique using the technique are well known, and further detailed description is omitted.
A set of feature points for which three-dimensional coordinates and normal vectors are calculated is hereinafter referred to as a three-dimensional point group. The three-dimensional model generation apparatus 200 stores the obtained three-dimensional point group data (step S13), and ends the three-dimensional point group generation process.
In the present embodiment, a three-dimensional point group is acquired from captured image data, but the following processing can also be performed using a three-dimensional point group obtained by another method such as laser measurement.

D. 3D model generation processing:
FIG. 3 is a flowchart of the three-dimensional model generation process. This is a process for generating a three-dimensional model representing a three-dimensional shape for each building based on the three-dimensional point cloud data described above. This process is realized by the gravity direction analysis unit 204, the point cloud separation processing unit 205, the building frame generation unit 206, the alignment processing unit 207, the polygon generation unit 208, and the texture generation unit 209 shown in FIG. It is. In terms of hardware, this is a process executed by the CPU of the three-dimensional model generation apparatus 200.

FIG. 3 shows the overall processing flow.
D1. Gravity direction analysis processing:
In the three-dimensional model generation process, a gravity direction analysis process is first performed (step S20). This is a process corresponding to the function of the gravity direction analysis unit 204, and is a process for analyzing the three-dimensional point group and specifying the gravity direction by the following procedure.
In the present embodiment, the principal component analysis of the entire three-dimensional point group is performed, and the third principal component is first set as the provisional gravity direction Gt. The reason why the direction of gravity can be specified by this method is as follows. The three-dimensional point group of the present embodiment is obtained by photographing a peripheral region centering on the target. Even if the filmed building is a high-rise building, its height is small compared to the horizontal extent of the filmed region. Therefore, the three-dimensional point group is distributed in a relatively flat region as shown in the drawing as a whole.
The principal component analysis is a method for obtaining a direction characterizing the distribution of the three-dimensional point cloud in the three-dimensional space. The first principal component, the second principal component, the third principal component are analyzed in order from the direction in which the distribution of the three-dimensional point cloud is large. Becomes the main component. For a three-dimensional point group distributed in a flat region, the first principal component and the second principal component are obtained based on the distribution in the horizontal direction, and the direction orthogonal thereto is the third principal component. Therefore, the third principal component represents the gravity direction Gt of the three-dimensional point group. However, since an error based on the distribution of the point cloud is included, the third principal component merely indicates the provisional gravity direction.
In this embodiment, since a three-dimensional point group distributed in a flat area is used, there is no need to perform complicated preprocessing such as extracting only a surface that is assumed to be in the vertical direction. It is possible to determine the direction of gravity.

  Since the three-dimensional point cloud of the embodiment has a normal vector together with the three-dimensional position coordinates, when the provisional gravitational direction is determined, the point cloud having the normal vector in a direction relatively along this is specified. can do. If such a point group is excluded, a point group constituting a horizontal plane can be excluded from point groups constituting a building, and a point group constituting a vertical plane of the building can be extracted. Further, if clustering is performed on the point group thus extracted based on the distance between the points and the normal direction, the point group can be separated in units of the side of the building. Then, if principal component analysis is performed for each surface and the third principal component is obtained, it becomes the normal vector direction of the side surface. The accuracy of this normal vector can be evaluated by the third eigenvalue. That is, the large third eigenvalue indicates that the variation of the point group is large in the normal direction, and indicates that the accuracy of the normal is low.

When the normal vectors for each side surface are obtained in this way, the gravitational direction G should be obtained by calculating the outer product Vi × Vj of the normal vectors Vi and Vj for any two side surfaces. However, since the normal vectors Vi and Vj each contain an error, the gravitational direction G thus obtained also contains an error. Therefore, in the embodiment, an arbitrary combination that is not parallel is selected from the obtained normal vectors, an individual gravity direction is obtained by an outer product, and a final gravity direction G is calculated by a weighted average thereof. The weight value in the weighted average can be, for example, the reciprocal of the sum (ei + ej) of the third eigenvalues representing the accuracy of the normal vectors Vi and Vj.
As described above, by performing statistical processing on the three-dimensional point group, it is possible to suppress the influence of errors included in the position coordinates and normal vectors of each point and determine the direction of gravity with high accuracy. It becomes possible.

D2. Point cloud separation processing:
Next, a point group separation process is performed (step S21). This is a process corresponding to the function of the point group separation processing unit 205, and is a process of separating a three-dimensional point group into building units.
In the course of this processing, first, the three-dimensional model generation apparatus 200 separates the three-dimensional point group for each hierarchy. Then, clustering of point groups is performed according to the distance between points in each hierarchy. Among these, since it can be determined that the roof surface has a top surface in a cluster that includes many normal vectors in the vertical direction, the cluster can be separated into those having the roof surface and those having only the side surface. . Then, from the cluster where the roof surface exists, the lower layer cluster in which the cluster is stacked is identified. This specification can be performed, for example, by determining that two-dimensional distances between clusters belonging to upper and lower hierarchies are equal to or less than a predetermined value as overlapping in the same building. In the case of a complex-shaped building that branches off on a higher floor, the upper branching portion is associated with a common cluster in the lower layer, so that the entire tree structure is recognized as one building.
By using the cluster association and the tree structure in this way, it is possible to separate a three-dimensional point group for each building, including specially shaped buildings.

D3. Building frame generation processing:
Next, building frame generation processing is performed (step S22). This is processing corresponding to the function of the building frame generation unit 206, and is processing for specifying the two-dimensional shape of the building.
In the course of this process, first, the three-dimensional model generation apparatus 200 extracts a side point group from the three-dimensional point group based on the normal vector. Then, a grid having a predetermined size is arranged along the separated side surfaces, and a point cloud is projected onto a horizontal plane to extract a lattice in which one or a plurality of points are present. By this processing, the number of points to be processed is reduced to the number of grid points at the position corresponding to the building frame. In this way, after reducing the number of points to be processed, the three-dimensional model generation apparatus 200 determines the building frame by thinning out the points determined to be arranged almost linearly.

D4. Alignment process:
Next, alignment processing is performed (step S23). This is a process corresponding to the function of the alignment processing unit 207, and is a process for adapting a three-dimensional point group to a two-dimensional map.
The alignment processing in the present embodiment does not perform fine correction of the shape and position in comparison with the two-dimensional map data for each building. In this embodiment, three-dimensional point clouds are obtained for the area around the target building, and these three-dimensional point clouds are relative to each other with the same accuracy as reproducing the top surface and side surface of each building. It should also represent a typical positional relationship. Under this concept, the present embodiment finds a shape and position conversion method that can most accurately fit the whole of the plurality of building frames obtained by the above processing to the two-dimensional map.
For such alignment, the correspondence between the vertices constituting the obtained building frame and the vertices on the two-dimensional map is tried in various combinations, and the three-dimensional point group is adapted to the two-dimensional map with the smallest deformation. Identify the corresponding relationship. For such identification, an annealing method or a simulated annealing method can be used.
According to this method, since the information of the three-dimensional point group can be sufficiently utilized for the shape of each building and the relative positional relationship between the buildings, it is possible to perform alignment with high accuracy.

D5. Polygon generation processing:
Next, polygon generation processing is performed (step S24). This is a process corresponding to the function of the polygon generation unit 208, and is a process of generating side and top polygons based on a three-dimensional point group for each building.
The three-dimensional model generation device 200 defines the upper surface based on the building frame obtained by the processing so far, and then translates it downward to set the polygon on the side of the building for each side. To do. Similarly, for a specially shaped building that diverges above, the side of the bifurcation part is generated by translating downward each upper surface, and the lower surface is translated downward. By doing so, it is possible to generate a low-level side polygon. In addition to using the shape of the upper surface as described above, the polygon may be defined based on the point group of the side surface separated by the building frame generation processing.

D6. Texture generation processing:
Finally, texture generation processing is performed (step S25), and a three-dimensional model for each building having positional accuracy that can be used as a three-dimensional map is generated.
4 to 6 are flowcharts of the texture generation process.
When the process is started, the three-dimensional model generation apparatus 200 generates a plain texture with a shape that matches the target polygon of the texture generation process (step S50). An example of generation is shown in the figure. Assume that the polygon in front of the building model shown on the left is the target polygon. As shown on the right side of the drawing, the three-dimensional model generation apparatus 200 generates a plain texture in a state where the target polygon is viewed from the front. The shape of the plain texture may be determined, for example, by converting the target polygon into a state seen from the front, or the aspect ratio of the target polygon may be obtained and a new rectangle corresponding to the aspect ratio may be generated. The number of pixels pix and piy of the plain texture is determined according to the shape. The target polygon itself obtained from any image in the plain texture may be used as the initial texture.

Next, the three-dimensional model generation apparatus 200 reads image data obtained by capturing the target polygon and performs a shielding determination (step S52). The processing method is shown in the figure. In this embodiment, since the feature is photographed from various directions above, there are a plurality of images in which the target polygon is captured. This process is performed for a plurality of images in which the target polygon is captured. However, it is not always necessary to target all images, and a predetermined number may be selected.
When the image is read, the three-dimensional model generation apparatus 200 cuts out the target polygon image and converts it into an image viewed from the front. This conversion can be realized by setting parameters for conversion to a state viewed from the front based on shooting conditions such as the shooting position, direction, and angle of view when the image is shot. In addition, when it is known that the target polygon is rectangular, conversion may be performed so that the image of the target polygon becomes rectangular without using the shooting condition. Furthermore, it is possible to adopt a method of converting the plain polygon obtained in step S50 so that the vertex of the target polygon matches.
In this way, when the image is converted into an image from the front, the three-dimensional model generation apparatus 200 performs a process of removing the shielding portion from the target polygon image. The already generated three-dimensional model can be used to determine which part is shielded. Based on the shooting conditions when each image is shot, the shooting is reproduced in the virtual space of the three-dimensional model including other features. In the image obtained in this way, it is relatively easy to identify which features are reflected based on the 3D model, so it is also possible to identify the parts that are covered by other features in the target polygon image. It is. By such a method, it is possible to remove the shielding part from the image of the target polygon.
The reading of the image data and the removal of the shielding portion may be performed at the time when the image data is read in this way, or may be performed at the time of the later-described processing for determining the color for each pixel. In this embodiment, the read image is converted into a front image, but it may be used without such conversion.

Next, the three-dimensional model generation apparatus 200 performs a process of determining the texture color for each pixel.
First, the three-dimensional model generation apparatus 200 selects a pixel to be processed (Step S54). In the drawing, an example is shown in which processing is performed in order as indicated by arrows a1 and a2 from the upper left pixel PP of the texture, but pixel selection is not limited to this order. For example, the feature points already obtained for the target polygon may be processed with priority.
When the pixel to be processed is selected, feature points around the pixel are extracted, and the relative position of the pixel to be processed is specified based on the feature points (step S56). In this embodiment, as the feature points, points corresponding to each image are obtained based on the image feature amount. In addition, vertices and other points obtained by image analysis of target polygon image data may be used as feature points.
The relative position specifying method is illustrated in the figure. It is assumed that feature points 1 to 4 exist around the processing target pixel PP. The relative positional relationship may be represented by, for example, a ratio of distances r1 to r4 from the target pixel PP to the feature points 1 to 4. The relative position is determined by the coordinate CPx of the target pixel PP on the axis connecting the feature points 2 and 3 and the coordinate CPy of the target pixel PP on the axis connecting the feature points w 1 with the lower left feature point 2 as a reference. You may make it represent a relationship. In addition to this, the relative positional relationship can be expressed by various methods.

When the relative positional relationship is obtained in this way, the three-dimensional model generation apparatus 200 specifies a corresponding pixel corresponding to the processing target pixel in each image data (step S58).
The image of processing was illustrated in the figure. The feature points 1 to 4 are points that exist in each image that is a material for texture generation. Therefore, if the feature points 1 to 4 are found in the image 1 and the relative positional relationship obtained previously is applied thereto, the corresponding pixel CP1 is obtained. Similarly, in another image 2, if the feature points 1 to 4 are found and the relative positional relationship is applied thereto, the corresponding pixel CP <b> 2 is obtained. As described above, if the same processing is performed on the image as the material for generating the texture, that is, each image read in S52, the corresponding pixel in each image can be specified.

The three-dimensional model generation apparatus 200 determines the color of the processing target pixel based on the average value of the corresponding pixels of each image obtained in this way (step S60). However, as described above in step S52, the occluded portion is removed from the image data. Therefore, this processing is performed using the color of the corresponding pixel excluding the shielding portion.
An example of processing is shown in the figure. The color of the processing target pixel PP1 can be obtained by averaging the colors of the corresponding pixels CP11, CP21, CP31 in the three images. The processing target pixel PP2 is obtained as an average value of the corresponding pixels CP12 and CP32 because the second corresponding pixel CP22 is removed by the shielding portion. The processing target pixel PP3 is obtained using the corresponding pixel CP33 because the two corresponding pixels CP13 and CP23 are removed by the shielding portion.
Various methods can be used as a method of determining a color based on a plurality of corresponding pixels. For example, the distribution of corresponding pixels for each color may be obtained and the peak value may be adopted.
The three-dimensional model generation apparatus 200 repeatedly executes the processing from step S54 until the colors are determined for all the pixels of the texture (step S62), stores the completed texture (step S64), and ends the texture generation processing. To do.

E. effect:
According to the above embodiment, since the color of the pixel to be processed is obtained by the average value of the corresponding pixels, if there is even one piece of image data that has not been removed by the shielding portion, the missing portion is supplemented and the processing target The color of the pixel can be determined. In addition, since processing is performed in units of pixels, a seam or the like when a plurality of images are combined like patchwork does not appear, and the image quality of the entire texture can be improved.
In the example,

The embodiment of the present invention has been described above. The present invention does not necessarily have all the functions of the above-described embodiments, and only a part may be realized. Moreover, you may provide an additional function in the content mentioned above.
Needless to say, the present invention is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present invention. For example, a part configured in hardware in the embodiment can be configured in software and vice versa.
In the embodiment, an example in which a three-dimensional point group is used has been described. However, texture generation is not necessarily limited to an aspect in which a three-dimensional point group is used.

  The present invention can be applied to generation of texture image data to be applied to a polygon of a three-dimensional model.

DESCRIPTION OF SYMBOLS 100 ... Shooting management apparatus 101 ... Camera 102 ... GPS
DESCRIPTION OF SYMBOLS 110 ... Shooting control part 112 ... Shooting sequence memory | storage part 120 ... Image recording part 200 ... Three-dimensional model production | generation apparatus 201 ... Image data memory | storage part 202 ... Three-dimensional point cloud data generation part 203 ... Three-dimensional point cloud data memory part 204 ... Gravity direction Analysis unit 205 ... Point group separation processing unit 206 ... Building frame generation unit 207 ... Alignment processing unit 208 ... Polygon generation unit 209 ... Texture generation unit 210 ... 2D map data 211 ... 3D model data

Claims (6)

A texture image data generation method for generating image data of a texture image to be applied to a polygon constituting a three-dimensional model of a feature by a computer,
As the steps executed by the computer,
(A) reading a plurality of image data obtained by photographing a surface corresponding to the polygon of the feature;
(B) For each pixel constituting the texture image, obtaining color information of a location corresponding to the pixel from the plurality of image data;
(C) generating the texture image by determining a color of the pixel based on the acquired color information ,
The step (B)
(B1) obtaining a correspondence relationship between a plurality of feature points in the plurality of image data;
(B2) identifying a location corresponding to the pixel on the basis of a plurality of feature points located around the pixel;
A method for generating texture image data. The method of generating texture image data according to claim 1 ,
The step (C)
A method of generating texture image data that determines a color of the pixel by excluding a portion of the image data that is shielded by other features. A method of generating texture image data according to claim 2 ,
The step (A) further reads a photographing condition of the image data,
The step (C) refers to a three-dimensional model representing the feature and a three-dimensional shape around the feature, and identifies the shielded portion based on the three-dimensional model and the imaging condition. A method for generating texture image data. A method for generating texture image data according to any one of claims 1 to 3 ,
The step (A) is a method of generating texture image data in which a surface corresponding to the polygon is converted into an image viewed from the normal direction of the surface, and the plurality of captured image data is read. A texture image data generation device for generating image data of a texture image to be applied to a polygon constituting a three-dimensional model of a feature,
An image data input unit that reads a plurality of image data obtained by photographing a surface corresponding to the polygon of the feature;
For each pixel constituting the texture image, obtaining color information of a location corresponding to the pixel from the plurality of image data, and determining the color of the pixel based on the obtained color information, A texture generation unit for generating a texture image ,
The texture generation unit
Obtaining a correspondence between a plurality of feature points in the plurality of image data;
A texture image data generation device that specifies a location corresponding to a pixel on the basis of a plurality of feature points located around the pixel . A computer program for causing a computer to realize a function of generating image data of a texture image applied to a polygon constituting a three-dimensional model of a feature,
(A) a function of reading a plurality of image data obtained by photographing a surface corresponding to the polygon of the feature;
(B) For each pixel constituting the texture image, a function of acquiring color information of a location corresponding to the pixel from the plurality of image data;
(C) causing the computer to realize the function of generating the texture image by determining the color of the pixel based on the acquired color information ;
As the function (B),
(B1) a function for obtaining a correspondence relationship between a plurality of feature points in the plurality of image data;
(B2) a function for specifying a location corresponding to the pixel on the basis of a plurality of feature points located around the pixel;
A computer program that Ru is realized.