JP2005063129A - Method, device and program for obtaining texture image from time-series image, and recording media for recording this program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain textures from time-series images without information on the relative positions and directions of a camera and a subject. <P>SOLUTION: A table creation mechanism 13 separates time-series images into a plurality of sequences. For each point within space information restored by a space information restoration mechanism 21, a feature point number in the space information, a sequence number, and information on the correlation of the feature point number with the sequences are recorded in a sequence table 14. A frame selection mechanism 11 refers to the sequence table 14 to obtain the coordinates on the image (frame) of each three-dimensional point constituting a three-dimensional plane, and based on this coordinate information selects a frame for obtaining textures. A texture coordinate determining mechanism 12 determines the coordinates on the image of each apex on the three-dimensional plane and cuts texture images out of the frame. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for acquiring a texture image from a time-series image, and more particularly to a method for efficiently acquiring a texture image in a system that restores a wide-area textured three-dimensional shape from a long time-series image.

  A technique for acquiring a three-dimensional point group from an image and acquiring a three-dimensional shape from the point group is known (see, for example, Non-Patent Document 1 and Non-Patent Document 2).

  With this technology, for example, as shown in Fig. 1, a 3D point cloud is acquired from an aerial image of a helicopter, and 3D shape data (city space model) of an urban area is constructed. (See Non-Patent Document 3). Since the video is composed of a sequence of images called frames taken every unit time, the video is hereinafter referred to as a time-series image.

  FIG. 2 is a diagram showing an example of aerial imaging of a building on the ground. As shown in FIG. 2, for example, an aerial image of the target building is taken at shooting times 011, 012, 013, 014, and 015 with a video camera mounted on a helicopter.

  FIG. 3 shows an example of a photographed time series image. As can be seen from FIG. 3, since the number of frames in which an object is projected is generally limited, when restoring a wide three-dimensional shape from a long time-series image, each partial image sequence (sequence) It is necessary to acquire a three-dimensional point cloud.

  Furthermore, since a large structure may span a plurality of sequences, it is necessary to restore a shape after integrating a plurality of point groups acquired from a plurality of consecutive sequences into one point group. In order to distinguish this integrated point cloud from the point cloud for each sequence, it is hereinafter referred to as spatial information.

  FIG. 4 is a diagram for explaining division from a time-series image into a fixed number of sequences. As shown in FIG. 4, for example, N frames are cut out from the time-series images of the target building group, and sequence 1 to sequence 4 are generated for each N frames. In the example shown in FIG. 4, each sequence is generated so that it overlaps a sequence followed by M frames.

  When restoring a 3D shape over a wide area from a long time-series image, a point cloud with 3D coordinate values is obtained for each sequence, the point cloud is integrated to generate spatial information, and the shape is restored from the spatial information. Do. Furthermore, since the shape is restored from the image, a texture image of the shape surface can be acquired.

  FIG. 5 shows a block diagram of the prior art. 2 is a texture image acquisition device including one or a plurality of CPUs and a memory, 21 is a spatial information recovery mechanism for recovering spatial information by acquiring a three-dimensional point cloud from a time-series image, and 22 is from spatial information. A shape restoration mechanism for restoring a three-dimensional shape, 23 a texture acquisition mechanism for obtaining a texture, 24 a file server, and 25 a texture storage device for storing a texture image. The file server 24 stores data such as time-series images, tracking coordinate values (to be described later), spatial information, and shape data.

  Hereinafter, a technique used in each component of FIG. 5 will be briefly described. As a method of acquiring a three-dimensional point group from each sequence in the spatial information restoration mechanism 21, there are methods such as “stereo image method” and “optical flow” (see Non-Patent Document 4). In any of the methods, it is necessary to establish the correspondence between feature points between images having different shooting directions, and the process of finding the correspondence between feature points in a time-series image is called “feature point tracking”.

  When tracking feature points, first, as shown in FIG. 6, one frame (for example, image13.tif at shooting time 013) is selected from the shot time-series images shown in FIG. 3, and four feature points are set. FIG. 7 shows how the feature points are tracked in the time-series images from the shooting time 011 to the shooting time 015. The result of the tracking process is stored as an image coordinate value in each frame of each feature point as shown in FIG. Hereinafter, these coordinate values are referred to as tracking coordinate values.

  The tracking coordinate value is stored for each sequence. For example, in FIG. tif to image15. The tracking coordinate values of the feature point 1 in each image up to tif are (300, 600), (300, 500), (300, 400), (300, 300), (300, 200).

  As a method for restoring the shape from the spatial information in the shape restoration mechanism 22, as shown in FIG. 9, a point set having the same height is found from the inputted spatial information, and the spatial information is converted into a point set having the same height. There has been proposed a method of forming a planar figure of a shape by dividing and forming a convex hull or circumscribed rectangle of the point set on a horizontal plane (see Non-Patent Document 1 and Non-Patent Document 5). A three-dimensional model is obtained from the planar figure of the shape.

As a texture acquisition method in the texture acquisition mechanism 23, in Non-Patent Document 6 and Non-Patent Document 7, a shape model is projected into an image using information on the position and orientation of a camera, and an object is detected from a time-series image. Shows how to select the frame (optimal frame) that is most visible in the center of the image and cut out the texture from that frame.
Yuji Ishikawa, Isao Miyagawa, Kaori Wakabayashi, Tomohiko Arikawa, “Reconstruction of shape model from 3D point set based on MDL criterion”, Proceedings of Image Recognition and Understanding Symposium (MIRU2002), vol. 165-170, Information Processing Society of Japan, 2002 Isao Miyagawa, Shigeru Nagai, Tomohiko Arikawa, “Reconstruction of spatial information by factorization with camera motion constrained”, IEICE Transactions, vol. J85-D-II, no. 5, pp. 898-966, 2002 Naotori Otani, Yuji Ishikawa, Yoshitaka Kuwata, Ushio Inoue, “Disaster prevention information provision system using 3D city maps”, Proceedings of the 10th Functional Graphic Information System Symposium, April 1999 Seiji Iguchi, Kosuke Sato, “3D image measurement”, Shosodo Yuji Ishikawa, Isao Miyagawa, Kaori Wakabayashi, Tomohiko Arikawa, “A Building Model Restoration Method from a 3D Feature Point Set Incorporating Line Constraints”, Information Processing Society of Japan (56th National Convention), 2003 Hiroshi Kawasaki, Tomoyuki Yatabe, Katsushi Ikeuchi, Masao Sakauchi, “Automatic Generation of 3D City from Omni Video Camera”, Computer Vision and Image Media, 119-4, 1999.11.17, Information Processing Society of Japan M. Uehara and H. Zen, "Building a Digital Town: From Digital Map to 3D Map by Mobile Observation", The Second International Symposium on Mized Reality (ISMR2001), March, 2001.

  In the conventional method, it is necessary to know the position and orientation of the camera when acquiring the texture. However, it is difficult to accurately acquire information such as camera position and posture. For example, it is necessary to attach a sensor for recording the position and orientation to the camera, and it is necessary to accurately record the position and orientation of the sensed information at each shooting time. Implementation of such means is expensive.

  Further, even if such information can be recorded, the time-series image is composed of a large number of frames, so that there is a problem that it takes time to select the optimum frame.

  The present invention solves the above-mentioned problems of the prior art, can acquire a texture from a time-series image without information on the relative position and orientation of the camera and the object, and the processing time required for acquiring the texture. It is an object to provide a technique capable of shortening.

  In order to solve the above problems, the texture image acquisition method of the present invention acquires a three-dimensional point group from a time-series image stored in a storage device, restores a three-dimensional shape from the three-dimensional point group, and A computer-aided texture image acquisition method for acquiring a texture image of a surface of a shape from an image included in the time-series image, the step of dividing the time-series image into a plurality of partial image sequences, and a feature in the partial image sequence A step of obtaining a coordinate value of the three-dimensional point group using coordinates on the image of the point, and a part used for obtaining the coordinate value of the three-dimensional point group for each point included in the three-dimensional point group A step of obtaining a correspondence relationship between the image sequence and the coordinate data on the image in the partial image sequence and storing it as correspondence relationship information, and each of the three-dimensional shapes constituting a three-dimensional surface from the stored correspondence relationship information point Acquiring a coordinate on the image, selecting an image from which the texture is acquired based on the acquired coordinate on the three-dimensional point image from the time-series image, and acquiring a texture from the selected image It is characterized by.

  Further, in the step of selecting an image for acquiring the texture from the time-series image, a centroid of each image is obtained based on coordinates on an image of a three-dimensional point constituting the surface of the three-dimensional shape, and a centroid of each image is obtained. Selects the image closest to the center of each image as the image for acquiring the texture.

  Further, in the step of acquiring the texture, for each vertex of the surface of the three-dimensional shape to which the texture is to be pasted, a coordinate on the image used when calculating the coordinate value of the three-dimensional point group is obtained, A texture image is cut out from the selected image based on the obtained coordinates on the image.

  Further, in the step of acquiring the texture, a translation, scale conversion, or rotation is performed based on the correspondence between the three-dimensional coordinates and the coordinates on the image of a point included in the surface of the three-dimensional shape to which the texture is to be pasted. Determining a composite transformation including movement, converting the three-dimensional coordinates of the vertices of the surface of the three-dimensional shape into coordinates on the image by the determined composite transformation, and selecting the selection based on the coordinates on the transformed image A texture image is cut out from the processed image.

  In addition, the process of converting the three-dimensional coordinates of the vertices of the surface of the three-dimensional shape into the coordinates on the image by the synthesis conversion is performed using 3 of the vertices of the surface of the three-dimensional shape to which the texture is to be pasted. It is characterized in that it is executed only for vertices for which there are no coordinates on the image used for obtaining the dimension coordinate value.

  The texture image acquisition device of the present invention includes spatial information restoration means for obtaining a three-dimensional point group from a time-series image, and shape restoration means for restoring a three-dimensional shape from the three-dimensional point group. In a texture image acquisition device for acquiring a texture image of a surface of a shape from an image included in the time series image, the time series image is divided into a plurality of partial image sequences, and the coordinates on the feature point image in the partial image sequence Is used to obtain the coordinate value of the three-dimensional point group, and for each point included in the three-dimensional point group, the partial image sequence used when obtaining the coordinate value of the three-dimensional point group, and the partial image Correspondence information creation means for obtaining a correspondence relationship between the coordinate data on the image in the sequence, each point included in the three-dimensional point group, the partial image sequence, and the coordinate data on the image in the partial image sequence Correspondence relationship that stores correspondence information The coordinates on the image of each three-dimensional point constituting the surface of the three-dimensional shape are acquired from the information storage means and the correspondence information storage means, and the texture is acquired based on the acquired coordinates on the image of the three-dimensional point. An image selection unit that selects an image from the time-series image, and a texture acquisition unit that acquires a texture from the image selected by the image selection unit.

  The present invention can also be realized by a computer and a software program, and the program can be recorded on a recording medium or provided through a network.

  According to the present invention, it is necessary to project a three-dimensional shape onto an image frame by using the image coordinates of feature points at the time of tracking processing when restoring the shape from the time-series image and acquiring the texture of the shape surface. There is no. Therefore, it is not necessary to know the relative position and orientation of the camera and the object, and the cost for acquiring and managing such information can be reduced.

  Also, the present invention creates a sequence correspondence table, so that when specifying a frame for texture acquisition, it is only necessary to search for a specific sequence rather than searching the entire time-series image. , Processing time required for texture acquisition can be shortened.

  The sequence correspondence table can be created only once, but since the frame search is performed for the number of faces to which the texture must be pasted, the effect of shortening the processing time becomes larger when the time-series image is long.

  FIG. 10 shows a configuration example of the present invention performed to solve the above problem. The texture image acquisition device 1 of the present invention excludes the texture acquisition mechanism 23 from the components of the conventional texture image acquisition device 2 shown in FIG. 5, and creates a correspondence table creation mechanism 13, a sequence correspondence table 14, a frame selection mechanism 11, texture coordinates. A decision mechanism 12 is added. Hereinafter, it demonstrates in order.

The correspondence table creation mechanism 13 includes, for each point P included in the spatial information, the sequence S used to acquire the three-dimensional coordinates of P and the tracking coordinate value data of the sequence S shown in FIG. Then, the points P ′ corresponding to the points P are respectively obtained and recorded in the sequence correspondence table 14. FIG. 8 shows that there are four feature points of a sequence S _i from 1 to 4, but in the correspondence table creation mechanism 13, when the three-dimensional coordinates of the point P are obtained from the sequence S _i , P 1 to 4 is identified and recorded in the sequence correspondence table 14.

  The frame selection mechanism 11 specifies a point set Q in the spatial information included in the surface to which the texture is pasted. Next, the tracking coordinate values of the points included in the point set Q are acquired from the file server 24 using the sequence correspondence table 14, and the image coordinates of each frame are obtained. A frame F for acquiring a texture is selected from the image coordinates. As shown in FIG. 7, there is a method of obtaining image coordinates at each frame in the sequence and selecting a frame in which the image coordinates are arranged at the center of the frame. According to this method, for example, the frame at the photographing time 013 shown in FIG. 7 is selected.

  The texture coordinate determination mechanism 12 specifies the vertex of the surface to which the texture is to be pasted from the point set Q, cuts out the texture image from the frame F using the image coordinates in the frame F, and records it in the texture storage device 25.

  If the vertex of the surface to which the texture is to be applied is not included in the point set Q, the combined transformation of rotational movement, translation, and scale conversion is performed between the three-dimensional coordinates of the point set Q and the image coordinates in the frame F. The three-dimensional coordinates of the vertices are converted into image coordinates by the generation. The generated composite transformation may include a part of rotational movement, parallel movement, and scale conversion.

  With the above device, the texture image can be acquired using the tracking coordinate value, so that it is not necessary to project the model shape onto the frame, and it is not necessary to know the position and orientation of the camera. In addition, when selecting a frame for acquiring a texture, the processing time is shortened because only a single sequence is searched instead of the entire time-series image.

  Hereinafter, embodiments of the present invention will be described in detail. In a video such as NTSC or HDTV, 30 frames are recorded as image data per second and stored in the file server 24. Time-series images are managed by storing a set of shooting time and frame file names. An example of management of time-series images in the file server 24 is shown in FIG.

  In the example of FIG. 11, the image file “image1.tif” corresponds to the shooting time 001, the image file “image2.tif” corresponds to the shooting time 002, and the image file “image2.tif” corresponds to the shooting time 003. The image file “image3.tif” is stored as time series image information, the image file “image4.tif” corresponding to the shooting time 004, and the image file “image5.tif” corresponding to the shooting time 005, respectively. ing.

  An example of tracking coordinate values for one sequence stored in the file server 24 is as shown in FIG. 8, and it is assumed that tracking coordinate values as shown in FIG. 8 are stored for each sequence. The spatial information is a sequence of three-dimensional coordinate values of each point, and numbers starting from 1 are assigned in order.

  The method of creating the sequence correspondence table 14 depends on how the three-dimensional point group acquired for each sequence is combined into one spatial information. In the embodiment of the present invention, the three-dimensional point group of each sequence is arranged in the same order as the order of the tracking coordinate values. Then, the coordinate values of the three-dimensional point group are connected in the order of the sequence start time, and a sequence of three-dimensional coordinate values of the spatial information is generated.

  An example of the sequence correspondence table 14 generated by the correspondence table creation mechanism 13 is shown in FIG. The sequence correspondence table 14 stores feature point numbers in spatial information, sequence numbers, and correspondence information of feature point numbers in sequences. The correspondence table creation mechanism 13 fills in the sequence number column and the feature point number column in the sequence in FIG. 12 to determine which feature point of each sequence each feature point in the spatial information is a sequence. Record in the correspondence table 14. In the sequence correspondence table 14, instead of the feature point numbers in the sequence, tracking coordinate value data of feature points shown in FIG. 8 may be recorded.

  An example of a sequence correspondence table generation process flow by the correspondence table creation mechanism 13 will be described with reference to FIG. First, the sequences are arranged in order of photographing start time (step S1), and the feature point number i of the spatial information and the sequence number j of the feature point are initialized to 1 (step S2).

In the sequence S _j , the number N of feature points that have been tracked to obtain a three-dimensional point cloud is acquired from the tracking coordinate value data of the sequence S _{j in} the file server 24 (step S3). Since the spatial information is data obtained by sequentially connecting the three-dimensional coordinate values of each sequence, the feature points i to i + N−1 of the spatial information are included in the same sequence S _j . Therefore, j is recorded in the sequence correspondence table 14 as the sequence number to which these points belong (step S4).

Further, since the three-dimensional coordinate values of the points included in the spatial information are arranged in the same order as the tracking coordinate values, the feature point numbers in the sequence S _{j with} respect to the feature points i to i + N−1 of the spatial information. 1 to N are recorded in the sequence correspondence table 14 (step S5). Thereafter, N is added to i and 1 is added to j (step S6). The above processing is repeated for all the sequences S _{1 to} S _M (step S7). When the above processing is performed for all the sequences, the processing is terminated and the sequence correspondence table 14 is output.

  In the above embodiment, the spatial information is configured by simply connecting the three-dimensional coordinate values acquired from each sequence. However, in consideration of convenience such as division and management of the spatial information, for example, a certain specific value is used. It is also conceivable to generate spatial information by arranging three-dimensional coordinate values along the coordinate axis.

  Even in such a case, if the processing performed when arranging the spatial information is known, the sequence correspondence table 14 can be configured by comparing the positions before and after the arrangement. Further, the correspondence table creation mechanism may be provided inside the spatial information restoration mechanism 21 so that the sequence correspondence table 14 may be created while maintaining the correspondence relationship before and after the array in the process of generating the spatial information. Even when the array processing is unknown, if the three-dimensional point group of each sequence is stored, it can be associated with the spatial information using the three-dimensional coordinate value as a key.

  Next, an example of a frame selection process flow by the frame selection mechanism 11 will be described with reference to FIG. In this embodiment, as shown in FIGS. 2 and 3, the texture is applied to the shape restored from the aerial image, and the side surface is hardly reflected in the image. The surface to be attached.

  As shown in FIG. 9, when restoring a three-dimensional shape from spatial information, a subset of points having the same height is found to form the upper surface. Therefore, it is easy to specify the set of points included in the surface by preserving this formation process. If information about the formation process is not stored, the points that the surface contains on the horizontal plane may be collected.

As processing of the frame selection mechanism 11, processing according to the flowchart shown in FIG. 14 is performed on each surface A to which the texture is pasted. First, the points of the spatial information included in the surface A to which the texture is pasted are listed and set as Q (= {q _i }) (step S11).

The sequence to which q _i belongs is obtained from the sequence correspondence table 14. As shown in FIG. 15, there is a case where one ground structure exists over a plurality of sequences of imaging regions, and as a result, the upper surface is composed of feature points of a plurality of sequences. In such a case, in the present embodiment, the number of the sequence to which the most points belong is set to jmax, and a texture acquisition frame is selected from the sequence S _jmax including the most points (step S12). Only the points belonging to the sequence S _jmax are left in the point set Q, and other points are deleted (step S13).

Next, the sequence by the correspondence table 14 obtains the feature point number k in the sequence S _jmax of the q _i (step S14), and acquires the image coordinates of the sequence S _jmax from the file server 24, each frame of the sequence S _jmax f The image coordinates of each q _i are obtained (step S15).

The center of gravity Gf is obtained from the image coordinates of {q _i } of each frame f, and the frame having the center of gravity Gf closest to the center of the frame is selected as the upper texture acquisition frame (step S16). By selecting a frame in which a large number of feature points are distributed in the vicinity of the center of the image in this way, it is possible to avoid a state where the upper surface protrudes from the frame.

  FIG. 16 shows an example of processing for selecting a texture acquisition frame for the upper surface of the ground structure shown in FIG. 15 by using the frame selection processing by the frame selection mechanism 11 described above. FIG. 15A shows three-dimensional points (characteristic points of the sequence) included in the XY plane to which the upper surface of the ground structure shown in FIG. 15B belongs.

The three-dimensional point shown in FIG. 15A includes a three-dimensional point acquired from the sequence S _j and a three-dimensional point acquired from the sequence S _{j + 1} . Since there are more feature points of the sequence S _{j + 1 on} the upper surface of the ground structure shown in FIG. 15A, the subsequent processing is advanced only by the points of the sequence S _{j + 1} .

The sequence S _{j + 1} is assumed to be composed of 21 frames from the shooting time 020 to the shooting time 040. The image coordinates of the feature points are obtained in each frame, and the center of gravity indicated by a double circle in FIG. 16 is calculated. As a result, the frame at the photographing time 029 at which the calculated distance between the center of gravity and the center of the frame is the shortest is selected as the optimum frame for acquiring the texture.

  With the above procedure, the texture acquisition frame is determined for each surface to which the texture is to be pasted. Finally, the texture coordinate determination mechanism 12 cuts out the texture image.

  An example of a texture coordinate determination process flow by the texture coordinate determination mechanism 12 is shown in FIG. The procedure of the texture coordinate determination mechanism 12 is executed for each surface A to which the texture is pasted. First, assume that the frame assigned to the surface A as a result of the processing by the frame selection mechanism 11 is F (step S21).

Similarly to the frame selection mechanism 11, the points q _{i of the} spatial information included in the plane A are collected and set as Q (= {q _i }) (step S22). With reference to the sequence correspondence table 14, if the sequence to which the point q _i belongs includes the frame F, the image coordinate r _i of the point q _{i in} the frame F is obtained (step S23). The point q _i for which the image coordinates in the frame F cannot be obtained is removed from the point set Q.

  If the image coordinates of all the vertices of the surface A are obtained in the steps so far (step S24), the process proceeds to step S30, where a partial image is cut out from the frame F as shown in FIG. Can be earned.

  In step S30, as shown in FIG. 18, a minimum partial image (image in a range surrounded by a dotted line) including a set of image coordinates of the vertexes of the surface A is acquired from the frame F, and is used as a texture image of the surface A. cut.

  If there is a point on the surface A where the image coordinates on the frame F cannot be obtained, as shown in FIG. 19, the three-dimensional coordinate system (XYZ) of the spatial information changes to the image coordinate system on the frame F ( xy) is generated, and the function T is used to obtain image coordinates on the frame F with respect to the vertex of the surface A.

For example, as shown in FIG. 19, a point (X ₁ , Y ₁ , Z _ave ) on the spatial information three-dimensional coordinate system (XYZ) is converted into an image coordinate system (x ₁ , y ₁ ) on the frame F. A function T is generated, and the generated function T is used to calculate the image coordinates (a ₁ , b ₁ ) on the frame F of the vertex (A ₁ , B ₁ , Z ₀ ) of the surface A in the coordinate system of the spatial information. obtain. Z _ave is an average value of coordinate values Z of points on the surface A.

  As shown in FIG. 19, if the shape of the surface is not the convex hull of the point set but the minimum circumscribed rectangle, the vertex is often not included in the original point set. Further, as shown in FIG. 15, vertices may be obtained from different sequences, and therefore image coordinates of all vertices are not necessarily obtained on a single frame F. Therefore, a means for obtaining image coordinates from the three-dimensional coordinates of the vertices is required.

In the present embodiment, it is assumed that the aerial image is stably captured immediately below, and the XY plane of the three-dimensional coordinate system of the spatial information is parallel to the xy plane of the image coordinate system. Therefore, the function T is a conversion function of two-dimensional coordinates, and a point on the XY plane is converted to a point on the xy plane by a combined transformation of translation (vector t), scale transformation (scale s), and rotational movement (rotation angle θ). Convert to point. For the points in the plane A, the points on the XY plane and the xy plane are set as q ′ _i and r _i , respectively, and t, s, and θ are obtained in the following order (step S25).
“Parallel movement”: The centers of gravity of {q ′ _i } and {r _i } are G and g, respectively, and t = (tx, ty) = g−G (step S26).
“Scale conversion”: V _i = q ′ _i −G and v _i = r _i −g are obtained for all points, and the average value of ‖v _i ‖ / ‖V _iと す る is defined as s (step S27).
- "rotation movement": For all the points, determine the rotation angle θ _i from V _i to v _i, and the average value θ (step S28).

Since the plane A has already been obtained, if the coordinates of its vertices are (A _j , B _j , C _j ), the coordinates in the XY plane are (A _j , B _j ). By substituting these coordinates into the following formula (1), the image coordinates (a _j , b _j ) of each vertex are obtained (step S29).

When the vertex image coordinates are obtained as described above, the minimum partial image including the set of vertex image coordinates (a _j , b _j ) is obtained from the frame F in the same manner as in FIG. , The texture image is saved as a texture image of surface A (step S30), and a texture image can be acquired.

In the present embodiment, the conversion between parallel planes (the XY plane and the xy plane) is considered in consideration of the aerial image captured directly below. However, even in the case of a more complicated positional relationship, the parameter Assuming a conversion function with an increase in the number, if there is more correspondence between the three-dimensional coordinate value and the image coordinate value than the number of parameters, the conversion function can be determined by the principle of solving simultaneous equations.
The present invention can be used when restoring a textured three-dimensional model from a time-series image obtained as a result of photographing a wide range of objects as a long sequence of images. Specific examples include the following target fields.
-Build a 3D model of a wide-area urban area from aerial images and use it for city planning and disaster simulation.
-Take a picture of the cityscape from the camera mounted on the vehicle, and create a textured 3D model of the building street. The obtained model data can be used for car navigation.

  In the application fields as described above, the position and orientation of the camera are generally acquired by GPS or a gyro, but (1) GPS satellite reception may not be successful. (2) Sensor position and camera position. And (3) the equipment is expensive. The method of the present invention using only image data is effective in solving such a problem.

It is a figure which shows the usage example of the three-dimensional shape acquisition technique from an image | video. It is a figure which shows the example of the aerial photography of the building on the ground. It is a figure which shows the image | photographed time series image. It is a figure explaining division | segmentation into a fixed number of sequences from a time-sequential image. It is a block block diagram of a prior art. It is a figure which shows the example of a setting of a feature point. It is a figure explaining the process which tracks a feature point. It is a figure which shows the example of the tracking result (tracking coordinate value) of a feature point. It is a figure which shows the shape restoration method from spatial information. It is a block diagram which shows the structural example of this invention. It is a figure which shows the example of the preservation | save format of a time series image. It is a figure which shows the example of a sequence corresponding table. It is a figure which shows an example of a sequence corresponding table production | generation processing flow. It is a figure which shows an example of a frame selection processing flow. It is a figure which shows the example of the decompression | restoration object which straddles two sequences. It is a figure which shows the example of the process which selects the frame for texture acquisition. It is a figure which shows an example of a texture coordinate determination processing flow. It is a figure which shows the example of the cutting-out process of a texture image. It is explanatory drawing of conversion of the three-dimensional coordinate system of space information, and an image coordinate system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 2 Texture image acquisition apparatus 11 Frame selection mechanism 12 Texture coordinate determination mechanism 13 Correspondence table creation mechanism 14 Sequence correspondence table 21 Spatial information restoration mechanism 22 Shape restoration mechanism 23 Texture acquisition mechanism 24 File server 25 Texture storage apparatus

Claims (8)

A three-dimensional point cloud is acquired from the time-series image stored in the storage device, a three-dimensional shape is restored from the three-dimensional point group, and a texture image of the surface of the three-dimensional shape is obtained from the image included in the time-series image. A method of acquiring a texture image by a computer,
Dividing the time-series image into a plurality of partial image sequences;
Obtaining a coordinate value of a three-dimensional point group using coordinates on an image of a feature point in the partial image sequence;
For each point included in the three-dimensional point group, the correspondence between the partial image sequence used to obtain the coordinate value of the three-dimensional point group and the coordinate data on the image in the partial image sequence is as follows. Determining and storing as correspondence information;
Obtaining the coordinates on the image of each three-dimensional point constituting the surface of the three-dimensional shape from the stored correspondence relationship information, and obtaining the texture based on the obtained coordinates on the image of the three-dimensional point in the time series Selecting from the image;
Obtaining a texture from the selected image. A method for obtaining a texture image. In the texture image acquisition method according to claim 1,
In the step of selecting an image for acquiring the texture from the time-series image,
The center of gravity of each image is obtained based on the coordinates on the image of the three-dimensional point constituting the surface of the three-dimensional shape, and the image whose center of gravity is closest to the center of each image is selected as the image for acquiring the texture. A texture image acquisition method characterized by the above. In the texture image acquisition method according to claim 1 or 2,
In the step of acquiring the texture,
For each vertex of the surface of the three-dimensional shape to which the texture is to be pasted, obtain the coordinates on the image used when obtaining the coordinate value of the three-dimensional point group,
A texture image acquisition method comprising cutting out a texture image from the selected image based on the obtained coordinates on the image. In the texture image acquisition method according to claim 1 or 2,
In the step of acquiring the texture,
From the correspondence between the three-dimensional coordinates of the points included in the surface of the three-dimensional shape to which the texture is to be pasted and the coordinates on the image, a translation or a scale conversion or a combined transformation including a rotational movement is determined.
By the determined composite conversion, the three-dimensional coordinates of the vertices of the surface of the three-dimensional shape are converted into coordinates on the image,
A texture image acquisition method, wherein a texture image is cut out from the selected image based on the coordinates on the converted image. In the texture image acquisition method according to claim 4,
The process of converting the three-dimensional coordinates of the vertices of the surface of the three-dimensional shape into the coordinates on the image by the composite conversion is the three-dimensional coordinates among the vertices of the surface of the three-dimensional shape to which the texture is to be pasted. A texture image acquisition method characterized in that it is executed only for vertices that do not have coordinates on the image used to determine the value. Spatial information restoration means for acquiring a three-dimensional point group from a time-series image; and shape restoration means for restoring a three-dimensional shape from the three-dimensional point group; and a texture image of the surface of the three-dimensional shape is converted into the time-series image. In the texture image acquisition device that acquires from the images included in
The time-series image is divided into a plurality of partial image sequences, and the coordinate values of the three-dimensional point group are obtained using the coordinates on the feature point images in the partial image sequence, and each point included in the three-dimensional point group is obtained. On the other hand, a partial image sequence used when obtaining the coordinate values of the three-dimensional point group, and correspondence information creating means for obtaining a correspondence relationship with the coordinate data on the image in the partial image sequence;
Correspondence information storage means for storing information on correspondence between each point included in the three-dimensional point group, the partial image sequence, and coordinate data on an image in the partial image sequence;
Obtaining the coordinates on the image of each three-dimensional point constituting the surface of the three-dimensional shape from the correspondence relationship information storage means, and obtaining an image for acquiring a texture based on the obtained coordinates on the image of the three-dimensional point in the time series Image selection means for selecting from images;
A texture acquisition unit comprising: a texture acquisition unit that acquires a texture from the image selected by the image selection unit.   A texture image acquisition program for causing a computer to execute the texture image acquisition method according to any one of claims 1 to 5.   A recording medium on which a texture image acquisition program for causing a computer to execute the texture image acquisition method according to claim 1 is recorded.

Images