JP2005332177A - Three-dimensional information processor, three-dimensional information processing method, and three-dimensional information processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automate the setting work of corresponding orienting points in a paired stereo picture. <P>SOLUTION: An object being a three-dimensional model is photographed in a moving picture while changing a photographing position. The orienting point is automatically set on the object on a first frame with the photographed moving picture data. The automatic set is done by extracting a feature point on the object by edge detection, for example. The set orienting point is tracked on each frame constituting the moving picture data to set the orienting point on each frame. Now, two frames are selected as the paired stereo picture from the orienting point-set frames; conducting photogrammetric analysis. Finally, the three-dimensional model of the object is generated by using three-dimensional information obtained from a result of the photogrammetric analysis. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a three-dimensional information processing apparatus, a three-dimensional information processing method, and a three-dimensional information processing program. For example, the present invention relates to one that generates a three-dimensional model of a subject from moving image data using photogrammetry technology.

The theory of photogrammetry has been studied for a long time, and the results are widely used in the field of civil engineering.
Photogrammetry is a technique for calculating three-dimensional information of a subject from two stereo pair images obtained by photographing the subject from different angles.

Hereinafter, the outline of the photogrammetry will be described with reference to FIG.
In photogrammetry, the subject 100 is photographed from two different directions by the camera 102 as shown in FIG.
As a result, as shown in FIG. 14B, a stereo pair image including a left image obtained by photographing the subject from the left side and a right image obtained by photographing from the right side is obtained.

Using the stereo pair image thus obtained, the operator recognizes corresponding points (hereinafter referred to as orientation points) in the left and right images. In the figure, it is recognized that the orientation point H1 of the left image and the orientation point H1 of the right image correspond to each other, and similarly, the correspondence relationship of other orientation points is recognized. The left image orientation point H5 and the right image orientation point H8 are blind spots of the subject and have no corresponding orientation points.
Next, the operator sets a plane coordinate system for the left image and the right image, and measures the coordinate values in the left image and the coordinate values in the right image of the corresponding orientation points in the left and right images.

Then, by applying photogrammetry theory (consisting of internal orientation, mutual orientation, absolute orientation, etc.) to these measurement values, the three-dimensional coordinate values of these orientation points can be calculated.
In addition, the control point H5 and the control point H8 that do not have a corresponding control point can be calculated by applying the parameters obtained as a result of the photogrammetry to the two-dimensional coordinate values in the captured image. it can.
Using these calculated three-dimensional coordinate values, a three-dimensional model 100a of the subject 100 as shown in FIG. 14C can be generated.

  By the way, in order to perform photogrammetry, a high level of skill is required, but a digital image of a subject is taken using a digital camera that has recently become widespread, and this is processed by a computer to process 3 of the subject. A technique for generating a dimensional model has also been developed. This technique is described in the following patent document.

JP 2002-31527 A

  In this technique, an algorithm capable of obtaining a result comparable to that of a skilled technician is programmed, and even a general user who does not have a skilled skill can easily perform photogrammetry with high accuracy.

In the above technique, most of the processes are automated, but the work of identifying the corresponding orientation points of the left and right images is still performed manually by the user.
That is, the work for associating the orientation points of the left image with the orientation points of the right image requires the user to input one by one into the computer.
In addition, when the number of control points is small, it is possible to perform manual input. For example, when generating a three-dimensional model of a subject having a complicated shape, or shooting a city landscape to capture a three-dimensional model of the city When many control points are set as in the case of computer graphics, manual input is difficult.

  Accordingly, an object of the present invention is to automate the work of setting the corresponding ground control points.

In order to achieve the above object, according to the first aspect of the present invention, in the first aspect of the present invention, from the acquired moving image data, moving image data acquiring means for acquiring moving image data obtained by shooting a subject while changing the shooting position, A first still image acquisition unit that acquires a first still image in which an orientation point is set on a subject and tracking the movement of the established orientation point using the moving image data, thereby setting the set orientation point. Corresponding image acquisition means for acquiring a second still image having a control point corresponding to a fixed point, a two-dimensional coordinate value of the control point on the first still image, and a standard on the second still image There is provided a three-dimensional information processing apparatus comprising: a three-dimensional coordinate value calculating means for calculating a three-dimensional coordinate value of a ground control point on the subject from a two-dimensional coordinate value of a fixed point.
In the invention according to claim 2, the image processing apparatus further includes a control point setting unit that acquires a still image from the moving image data, and sets a control point on the subject with the still image, wherein the first still image acquisition unit includes the first still image acquisition unit, 3. The still image in which an orientation point is set, or a still image in which an orientation point is tracked from the still image by using the moving image data is acquired as a first still image. A dimensional information processing apparatus is provided.
In the invention according to claim 3, the measured values of at least two plane positions and the measured values of the height of at least three points are known among the orientation points of the first still image or the second still image. A reference point receiving unit that receives designation of a reference point, and an actual value input unit that receives an input of an actual measurement value of the received reference point, and the three-dimensional coordinate value calculation unit includes an actual measurement of the reference point. 3. The three-dimensional information processing apparatus according to claim 1, wherein an actual size three-dimensional coordinate value is calculated using the value.
In the invention according to claim 4, still image acquisition means for acquiring two still images having corresponding control points from the moving image data by tracking a control point set for a subject, and the two acquired still images Evaluation means for evaluating the accuracy of the three-dimensional coordinate value of the subject calculated from the orientation point from the image, wherein the first still image acquisition means and the corresponding image acquisition means are the evaluation means. The three-dimensional information processing apparatus according to claim 1, 2 or 3, wherein a still image evaluated to have a predetermined accuracy is acquired.
The invention according to claim 5 further comprises a control point resetting means for newly setting a control point in the second still image so as to overlap with at least three control points of the second still image. The corresponding image acquisition means acquires the third still image and the fourth still image having the resetting control point by tracking the movement of the resetting control point with the moving image data, The three-dimensional coordinate value calculation means is configured to obtain a control point on the subject from the two-dimensional coordinate value of the control point on the third still image and the two-dimensional coordinate value of the control point on the fourth still image. The three-dimensional coordinate value is calculated, the three-dimensional coordinate axis calculated from the first still image and the second still image, and the three-dimensional calculated from the third still image and the fourth still image. The overlapping orientation generated so that the coordinate axes coincide with each other. Providing a three-dimensional information processing apparatus according to any one of claims of claims 1, characterized by comprising a coordinate transforming means for performing coordinate conversion to claim 4 with reference to the coordinate values.
According to a sixth aspect of the present invention, in a computer comprising moving image data acquisition means, first still image acquisition means, corresponding image acquisition means, and three-dimensional coordinate value calculation means, the moving image data acquisition means A moving image data acquisition step of acquiring moving image data obtained by shooting the subject while changing the shooting position; and a first control point set for the subject from the acquired moving image data by the first still image acquisition means. The first still image acquisition step of acquiring a still image of the image, and the corresponding image acquisition means tracks the movement of the set orientation point using the moving image data, thereby corresponding to the set orientation point. A corresponding image acquisition step for acquiring a second still image having a control point to be detected, and a three-dimensional coordinate value calculation means for detecting a target on the first still image. A three-dimensional coordinate value calculating step for calculating a three-dimensional coordinate value of the orientation point on the subject from the two-dimensional coordinate value of the point and the two-dimensional coordinate value of the orientation point on the second still image; A three-dimensional information processing method is provided.
According to the seventh aspect of the present invention, a moving image data acquisition function for acquiring moving image data obtained by photographing a subject while changing a photographing position, and a first stationary point in which a ground control point is set for the subject from the acquired moving image data. A first still image acquisition function for acquiring an image, and a second still image having a control point corresponding to the set control point by tracking the movement of the set control point using the moving image data. From the corresponding image acquisition function for acquiring an image, the two-dimensional coordinate value of the orientation point on the first still image, and the two-dimensional coordinate value of the orientation point on the second still image, A three-dimensional information processing program for realizing a three-dimensional coordinate value calculation function for calculating a three-dimensional coordinate value of an orientation point by a computer is provided.

  According to the present invention, a three-dimensional model can be automatically generated from a moving image obtained by photographing a subject.

(Outline of the embodiment)
A moving subject is photographed while changing the photographing position of a subject to be a target of the three-dimensional model.
In the moving image data photographed in this way, a control point is automatically set on the subject in a frame that is a starting point for tracking a control point. This automatic setting is performed, for example, by extracting feature points on the subject by edge detection or the like.
The set orientation point is tracked in each frame constituting the moving image data, and the orientation point on each frame is set.
Then, two frames are selected as a stereo pair image from the frames in which these control points are set, and photogrammetric analysis is performed. Further, a three-dimensional model of the subject is generated using the three-dimensional information obtained as a result of the photogrammetry analysis.

Hereinafter, an outline of a procedure for generating a three-dimensional model according to the present embodiment will be described with reference to FIG.
FIG. 1A is a diagram for explaining a method of photographing the subject 2 using the video camera 3.
The video camera 3 is a device that takes a moving image of a subject. In the present embodiment, a commercially available digital video camera is used as the video camera 3.

  The digital video camera includes a light receiving surface on which CCD (Charge Coupled Device) elements are arranged, and projects a subject on the light receiving surface to convert an image into digital data. Then, the temporal change of the projected image is recorded as moving image data on a storage medium such as a magnetic tape or a magneto-optical disk.

  It is also possible to use an analog type as the video camera 3 or to record video on a film medium such as an 8 mm camera instead of the video camera 3. In this case, the video imaged later may be converted into digital data.

The subject 2 is a generation target of the three-dimensional model, and is an object having a certain three-dimensional structure. The subject 2 varies depending on the purpose of generating the three-dimensional model (surveying, creation of computer graphics).
Using the video camera 3 configured as described above, the user captures the subject 2 while moving around the subject 2 as shown in FIG.

Next, the three-dimensional information processing apparatus (configuration will be described later) reads the moving image data shot in this way, and generates a three-dimensional model of the subject as follows.
The moving image data is composed of a series of frames (still images) taken while changing the direction of the subject 2, and the three-dimensional information processing apparatus selects the frame A therefrom. As the frame A, for example, the first frame constituting the moving image data can be adopted.

Then, the three-dimensional information processing apparatus extracts the subject 2 from the frame A using image processing such as edge detection, and extracts feature points (H1 to H7) having features such as corners and wrinkles of the subject 2, for example. This is set (registered) as an orientation point.
The three-dimensional information processing apparatus tracks the orientation points in the frame adjacent to the frame A, and further repeats the tracking in the adjacent frames to identify the orientation point in the frame B.
In each frame from frame A to frame B, the orientation points are set in association with each other by tracking, so that the three-dimensional information processing apparatus uses two frames from frame A to frame B as stereo pair images. select.
Then, the three-dimensional information processing apparatus performs photogrammetry using the orientation points shown in the two stereo pair images (for example, frame A and frame B).
By this process, the three-dimensional coordinate value of each orientation point is calculated, and the three-dimensional information processing apparatus generates a three-dimensional model 5a of the subject 2 using these coordinate values.

The three-dimensional information processing apparatus further continues the tracking of the orientation point and specifies the orientation point in the frame C. Similarly, the frame B and the frame C are adopted as the stereo pair image to generate the three-dimensional model 5b.
The three-dimensional information processing apparatus repeats the above processing to generate a plurality of three-dimensional models, and then connects these to generate a three-dimensional model 6. The connection can be performed if there are at least three common control points in the three-dimensional model (practically four points).
Thus, a three-dimensional model that covers the range captured by the video camera 3 can be generated.

(Details of the embodiment)
FIG. 2 is a block diagram showing a functional configuration of the three-dimensional information processing apparatus 1.
The three-dimensional information processing apparatus 1 can be realized, for example, by executing a three-dimensional information processing program on a personal computer.

The three-dimensional information processing apparatus 1 includes a moving image data acquisition unit 61 that acquires moving image data, a moving image data processing unit 62 that acquires control point information necessary for photogrammetry from the moving image data, and a control point information storage unit that stores control point information. 66, a three-dimensional model generation unit 67 that generates a three-dimensional model by performing photogrammetric analysis using the stored control point information, a three-dimensional model connection unit 68 that connects a plurality of generated three-dimensional models, and the like. ing.
Hereinafter, functions of these components will be described.

The moving image data acquisition unit 61 constitutes moving image data acquisition means, for example, receiving transfer of moving image data from a video camera, reading moving image data stored in a storage medium such as a magnetic tape, a magneto-optical disk, a hard disk, Alternatively, moving image data is acquired by receiving moving image data transmitted via a network such as the Internet.
The moving image data is configured so that still images called frames can be acquired in time series, and by referring to adjacent still images, it is possible to capture the movement of the subject that changes every moment.

The moving image data processing unit 62 receives moving image data from the moving image data acquisition unit 61, acquires frames from this, and arranges them in time series.
The control point setting unit 63, the reference point specifying unit 64, and the control point tracking unit 65 are used to set a control point, track a reference point, and set a pair of frames as a stereo pair image. Processing such as selection and acquisition of ground control point information from the stereo pair image is performed.

The control point setting unit 63 sets a control point for the subject in the start point frame that is the start point of the control point tracking. Also, during the tracking process, the orientation point is behind the subject and disappears in sequence, but the orientation point is generated again when the disappearance reaches about 30%.
In photogrammetry, it is generally necessary that the left and right images have the subject overlapped by 60% or more. However, in this embodiment, when the overlap falls below 70% with a margin, the standardization is performed again. Generate a fixed point.
The control point setting unit 63 constitutes a control point setting unit that acquires a still image (frame) from moving image data and sets a control point for a subject on the still image.

Here, the procedure in which the orientation point setting unit 63 sets the orientation point will be described with reference to FIG.
The ground control point setting unit 63 detects the edge of the subject 10 on the frame as shown in FIG.
Then, in order to extract the subject 10 from the background, image extraction by labeling processing is performed. The labeling process is a process for grouping pixels based on color information of pixels of an image.

The control point setting unit 63 sets the rectangular area 11 outside the extracted subject 10 so that the shortest distance between each side and the subject 10 (the number of pixels can be set as a unit) is A (FIG. 3). (B)). A is a distance that can be specified by the user, and an appropriate value is selected from the size of the subject. A portion included in the rectangular area 11 is a target for setting and tracking a control point.
Here, the reason why the rectangular area 11 has a margin of the distance A is that the disturbance of the edge due to the background image is taken into consideration, so that the subject is always included in the rectangular area 11.

After setting the rectangular area 11, the orientation point setting unit 63 generates a mesh having a side t of t in the rectangular area 11 as shown in FIG. t is a length that can be set by the user, and an appropriate value is selected from the size of the subject and the surveying accuracy. The mesh is used to track the ground control points across other frames.
In addition, when the mesh protrudes from the rectangular area 11, whichever may be prioritized, here, the mesh is prioritized.

After generating the mesh, the orientation point setting unit 63 detects feature points (such as corners and buttocks) of the subject by image processing, and sets them as orientation points.
FIG. 3D shows the location of the orientation points H <b> 1 to H <b> 7 at the feature points of the subject 10. The mesh is not shown.
Then, the orientation point setting unit 63 registers the shade information of the mesh located at each feature point as a tracking template (described later).

The reference point specifying unit 64 (FIG. 2) is a functional unit that receives a reference point specification from the user.
In photogrammetry, in order to determine the scale of the three-dimensional model, it is necessary to designate on the subject a point where at least two coordinate values in the vertical direction and three coordinate values in the horizontal direction are known. For this reason, three points whose three-dimensional coordinate values are known may be designated as reference points, but four points are practically designated.
Photogrammetry is possible without using a reference point, but when a reference point is not used with only a ground control point, a three-dimensional model similar to the subject can be obtained.

The reference point designating unit 64 displays the subject 10 together with the orientation point set by the orientation point setting unit 63 on a display or the like, and allows the user to select the orientation point used as the reference point.
In FIG. 3D, orientation points H1, H2, H4, and H5 are designated as reference points, and each orientation point also serves as the reference points K1, K2, K4, and K5. As described above, the reference point may be set as the orientation point as described above, or a point other than the orientation point may be designated.
The reference point specifying unit 64 allows the user to specify a reference point and input coordinate values in the real space.

For example, when the subject 10 is a rectangular parallelepiped having a side of 200 mm, it is assumed that the x axis is in the H1 to H2 direction, the y axis is in the H1 to H4 direction, and the −z axis is in the H1 to H5 direction. (0, 0, 0), K2 (200, 0, 0), K4 (0, 200, 0), K5 (0, 0, -200), etc. are input.
The above is an example, and the reference point and the coordinate axis can be selected easily.

In this way, the reference point designating unit 64 receives reference point designation means for accepting designation of reference points for which measured values of at least two plane positions and measured values of heights of at least three points are known. Measured value input means for receiving input of measured values of the reference points is provided.
Then, the reference point is traced to the left and right images (first still image and second still image) of the stereo pair image. For this reason, the reference point is designated in the first still image or the second still image. Accepted.

The control point tracking unit 65 (FIG. 2) is a functional unit that extracts information necessary for photogrammetry analysis from moving image data. (1) Tracking of control points and reference points, (2) Selection of stereo pair images, ( 3) It has a function of resetting control points and reference points, and (4) extracting control point information.
Hereinafter, these functions will be described.

(1) Tracking of control points / reference points The control point tracking unit 65 uses the control points and reference points set by the control point setting unit 63 and the reference point specifying unit 64 over a plurality of frames adjacent to the start point frame. Chase.
Various techniques for tracking the movement of points on a frame using moving image data have been put into practical use. For example, a method using matching using a template of mesh shading information and a vector of pixels represented by an optical flow There is a method of tracking from information.
Any technique may be used, but in the present embodiment, tracking of the orientation point / reference point is performed using a template of shading information.

FIG. 4 is a diagram for explaining a method of tracking a ground control point using a template of light and shade information.
As shown in the figure, the orientation point setting unit 63 creates a tracking template from the shade information of the mesh around the orientation point set in the starting frame.
In this template, the gray scale distribution of the orientation point and the surrounding mesh is recorded, and the orientation point tracking unit 65 matches this template with the mesh of the next frame adjacent to the starting frame.

In adjacent frames, since the amount of movement of the subject is small, it is considered that there is an area having a light and shade distribution similar to that of the template in the vicinity of the mesh where the tracking control point is present.
In this way, by matching templates with adjacent frames, it is possible to specify a mesh in which the orientation point is shown on this frame.
In this way, the orientation point tracking unit 65 identifies the location of the orientation point in the adjacent frame, and further repeatedly applies this processing to the adjacent frame, so that the orientation point set in the origin frame is plural. Can be tracked across frames.

In addition, when there are a plurality of different shades of pixels in one mesh, the average of these can be used as the shade of this mesh.
Further, a tracking template can be configured using color information and other information in addition to the grayscale information.
As described above, the orientation point tracking unit 65 includes tracking means for tracking the orientation point.

(2) Selection of stereo pair images Since the orientation points and reference points are tracked over each frame, even if two arbitrary frames are selected and combined as a stereo pair image, the orientation points and reference points on both frames are selected. A reference point can be made to correspond.
Therefore, it is possible to arbitrarily select two frames for constituting a stereo pair image from moving image data.

Various standards can be considered as the reference point tracking unit 65 for selecting a frame to be paired as a stereo pair image, but here, as an example, three-dimensional information calculated from a frame pair using vertical parallax is considered. The accuracy is evaluated, and a set of frames with high accuracy is selected as a stereo pair image.
More specifically, according to the theory of photogrammetry, each orientation point has a value called vertical parallax in a stereo pair image. Therefore, an appropriate value α for evaluating the accuracy of vertical parallax is set, and a frame in which the average value of vertical parallax at each orientation point is less than α is selected.
α may be set in advance by the orientation point tracking unit 65 or may be configured to be set by the user.

  This selection method is an example, and in addition to this, it may be configured to select when the orientation point at which the vertical parallax is α or more does not exceed a certain set value N, or the disappearance ratio of the orientation point is The frame can be configured to be selected using other evaluation criteria, such as selecting a frame that does not deteriorate the surveying accuracy.

An example of a frame selection method will be described with reference to FIG.
In this example, a case will be described in which two frames are selected from β + 1 consecutive frames starting from the i-th frame.
The i-th frame is a frame that is a starting point for the orientation point tracking. The orientation point is set (or reset) in this frame, and a tracking template is created.
β is a value that can be set by the user, and can be set according to the subject and the shooting situation of the moving image.

As shown in FIG. 5A, the orientation point tracking unit 65 first tracks the orientation point in the (i + 1) th frame.
Then, the vertical parallax at the (i + 1) th frame and the i-th frame is calculated and compared with α.
When the vertical parallax is less than α, the i + 1-th frame and the i-th frame are selected as a stereo pair image.

When the vertical parallax is equal to or greater than α, the orientation point tracking unit 65 further tracks the orientation point on the (i + 1) th frame in the i + 2th frame, as illustrated in FIG.
Then, the vertical parallax at the i + 2 frame and the i + 1 frame is calculated and compared with α.
When the vertical parallax is less than α, the i + 1-th frame and the i-th frame are selected as a stereo pair image.

When the vertical parallax is equal to or greater than α, the orientation point tracking unit 65 further determines the vertical parallax from the i + 2th frame and the i-th frame.
When the vertical parallax is less than α, the orientation point tracking unit 65 selects the i + 2th frame and the frame of the i scene as a stereo pair image.

The orientation point tracking unit 65 repeats the same processing, and as shown in FIG. 5C, tracks the orientation point in the i + k-th frame, the i + k-1th frame, the i + k-2th frame,. And a combination in which the vertical parallax is less than α is searched.
The orientation point tracking unit 65 searches for a combination of stereo pair images until a combination of frames in which the vertical parallax is less than α is found or k reaches β.

When a stereo pair image combination is obtained from the i-th frame to the i + β-th frame, or when k reaches β, the orientation point setting unit 63 generates a mesh again in the i + β-th frame and sets the orientation point. To do. Then, the orientation point tracking unit 65 places this frame as a new i-th frame, and similarly attempts to select the next β frames. The moving image data processing unit 62 repeats this operation until the end of the moving image data, and selects a plurality of sets of stereo pair images.
The above processing will be described later with reference to the flowchart of FIG.

  In this manner, the orientation point tracking unit 65 obtains the first still image from which the orientation point is set for the subject (the frame with the smaller frame number in the above example) from the moving image data. The second still image having a control point corresponding to the control point set in the first still image (in the above example, the frame is large) by tracking means and movement of these control points using moving image data Corresponding image acquisition means for acquiring the other frame).

  In addition, the orientation point tracking unit 65 obtains two still images having corresponding orientation points from the moving image data by tracking the orientation points set for the subject, and based on these two still images, An evaluation unit that evaluates the accuracy of the three-dimensional coordinate value of the subject calculated using the orientation point is provided, and a still image evaluated as having a predetermined accuracy by the evaluation unit is acquired as a stereo pair image.

(3) Resetting of the control point / reference point In this embodiment, since the subject is video-recorded while changing the shooting position, the control point becomes a shadow of the subject and gradually disappears as the frame advances. In addition, feature points that are hidden behind the subject appear newly.
Therefore, the orientation point tracking unit 65 tracks the orientation point and monitors the disappearance ratio of the orientation point, and causes the orientation point setting unit 63 to reset the orientation point again in a frame in which the orientation point has disappeared by 30%. . If necessary, the user is allowed to reset the reference point.

In this case, the orientation point setting unit 63 generates a mesh again after enclosing the subject with a rectangular area in the frame specified by the orientation point tracking unit 65. Then, an orientation point is set on the subject, and a tracking template is generated.
When resetting the control points in this way, the control point setting unit 63 sets the control points so as to include at least three control points (practically four points) tracked in the previous stage. Thus, the control point setting unit 63 constitutes a control point resetting unit.

Then, the orientation point tracking unit 65 designates at least 3 (practically 4 points) of orientation points tracked in the previous stage as new reference points in the frame where the orientation points are reset.
Since the coordinate values of the orientation points tracked in the previous stage are calculated from the reference points in the previous stage, these orientation points can be used as the reference points.

In addition to the disappearance of the orientation point, the orientation point may be reset. Hereinafter, these examples will be described.
As such a case, for example, a new subject may appear as the frame advances.

As shown in FIG. 6A, when a rectangular area 15a is set on the subject 15 and the orientation point is tracked, another subject 16 appears.
In this case, the moving image data processing unit 62 extracts the subject 16. Then, similarly to the subject 15, a rectangular area 16a is set around the subject 16 (FIG. 6B), and a mesh is generated. Then, an orientation point is set on the subject 16 and is tracked. The reference point of the subject 15 can be used for the photogrammetry processing of the subject 16.
Also, a reference point can be set for the subject 16 using the orientation point of the subject 15.

Further, as a case of resetting the orientation point, an important orientation point may be lost in the tracking process.
For example, as shown in FIG. 7A, it is assumed that the orientation points in the regions 22 and 23 disappear in the tracking process in the subject 20.
In this case, the moving image data processing unit 62 generates a rectangular area 20a around the subject 20, as shown in FIG. Then, a mesh is generated in the rectangular area 20a to reset the orientation point. Thereby, the lost orientation point can be set again.

(4) Extraction of ground control point information The ground control point tracking unit 65 uses the frame selected from the moving image data as a stereo pair image, and extracts ground control point information.
This process is performed by measuring the coordinate values of the corresponding orientation points in the frame selected as the left image and the frame selected as the right image.
The coordinate system for measuring the coordinate value of the orientation point is a two-dimensional coordinate system set on the frame. For example, the x axis in the horizontal direction and the y axis in the vertical direction can be set with the lower left corner of the frame as the origin. it can.
The number of pixels (number of pixels) is used as the measurement unit, and the pixel size of the light receiving unit of the video camera is used later, and is converted into millimeter units later. This is possible if the vertical and horizontal dimensions of the light receiving surface and the vertical and horizontal numbers of pixels are known.

FIG. 8 is a table showing an example of the control point information extracted by the control point tracking unit 65.
In FIG. 8, the i-th frame is represented as F (i) or the like. As shown in FIG. 8, the control point information is acquired for each stereo pair image.
For example, in the stereo pair image specified by the stereo pair number 1, F (20) is selected as the left image and F (56) is selected as the right image.

And the coordinate value in the left image of the orientation point H1 is (282, 333), and the coordinate value in the right image is (526, 623). Similarly, the coordinate value in the left image and the coordinate value in the right image are measured up to the orientation point H303.
However, the orientation points H4 and H303 disappear in F (56). The orientation point H2 is set as the reference point K2.

In the stereo pair image specified by the stereo pair number 2, F (105) is selected as the left image, and F (132) is selected as the right image.
Then, the coordinate values of the orientation points H1 to H303 are measured. In addition, the coordinate value of the lost orientation point is blank.
Similarly, the coordinate values of the orientation points are measured for the stereo pair images specified by the stereo pair image numbers 3, 4,.
As orientation information, in addition to the coordinate values of orientation points in these left and right images, when orientation points are reset, the orientation points that are duplicated before and after resetting can be identified. The spatial connection relationship of can be understood.

Returning to FIG. 2, the moving image data processing unit 62 outputs the measurement values measured in each stereo pair image as described above to the orientation point information holding unit 66 as orientation point information. The control point information holding unit 66 stores and holds this.
Thereby, the control point information regarding the plurality of stereo pair images is accumulated in the control point information holding unit 66.

The three-dimensional model generation unit 67 performs photogrammetry analysis for each stereo pair image using the control point information stored in the control point information holding unit 66, and generates a three-dimensional model for each stereo pair image. .
Here, the procedure of the photogrammetry processing will be described with reference to the flowchart of FIG. This process is performed for each stereo pair image.

The three-dimensional model generation unit 67 selects one of the plurality of stereo pair images generated by the moving image data processing unit 62 and acquires the control point information of the stereo pair image from the control point information holding unit 66 ( Stereo pair image 305).
Next, the three-dimensional model generation unit 67 internalizes the acquired orientation point information (step 310).
Internal orientation is a process of converting a coordinate value to be used in later internal orientation, for example, by correcting distortion when a subject is projected onto a light receiving surface of a video camera. Conversion from pixel units to millimeters is also done with internal orientation.

The three-dimensional model generation unit 67 internally evaluates the coordinate values of the orientation points, and then mutually evaluates them (step 315).
The relative orientation is an operation for obtaining a relative inclination or positional relationship between stereo pair images. A three-dimensional model similar to the subject can be obtained by relative orientation.

Next, the three-dimensional model generation unit 67 performs absolute orientation using the result of mutual orientation (step 320).
Absolute orientation is an operation for obtaining a scale of a three-dimensional model similar to a subject obtained by mutual orientation using a reference point in a stereo pair image, and is sometimes called external orientation.
By the absolute orientation, the coordinate value in the real space of each orientation point can be obtained. In addition, even in the orientation point disappeared in any one of the frames, it is possible to calculate the coordinate value in the real space from the coordinate value in the other frame using the absolute orientation result.

Since the absolute orientation determines the coordinate value of each orientation point in real space, it is possible to generate a three-dimensional model of the subject and the actual size.
For example, when the calculated control points are connected by line segments and displayed, a wire frame model of the subject can be obtained. A surface model of the subject can be obtained by extending the surface using the orientation points.
As described above, the three-dimensional model generation unit 67 generates a three-dimensional model for each stereo pair image.
As described above, the three-dimensional model generation unit 67 uses the two-dimensional coordinate value of the orientation point on the first still image constituting the stereo pair image and the two-dimensional coordinate value of the orientation point on the second still image. 3D coordinate value calculating means for calculating the 3D coordinate value of the orientation point on the subject is provided.

Returning to FIG. 2, the 3D model connecting unit 68 is a functional unit that connects the 3D models generated by the 3D model generating unit 67.
In the three-dimensional model generated from one set of stereo pair images, the part that becomes the blind spot of the subject is lost.
Therefore, by connecting a three-dimensional model based on a stereo pair image obtained by photographing the subject from another direction, it is possible to generate a three-dimensional model of a part that has become a blind spot.

When two three-dimensional models are connected, at least three common points (practically four) are required in common. As the common point, a common orientation point or reference point common to both three-dimensional models can be used.
By using these common points, the relative positional relationship and rotational angle relationship between the spatial coordinate axes of both three-dimensional models can be known. Therefore, it is possible to connect the three-dimensional models by performing coordinate transformation using these relationships so that the spatial coordinate axes of the three-dimensional models coincide.

  In this embodiment, since the orientation point is tracked by the frame, three common points can be found in both three-dimensional models. That is, when a three-dimensional model (model 1) generated from a certain stereo pair image and a three-dimensional model (model 2) generated from a stereo pair image tracked after that stereo pair image are considered, This is because the fixed point also has the model 1 (when the control point is not reset).

  In addition, even if the control point is reset, in order to generate a new control point so that it overlaps with the control point in the previous stage, the three-dimensional model generated from the stereo pair image before resetting and after resetting The three-dimensional model generated from the stereo pair images can be connected using the orientation points generated in duplicate.

Here, the connection of the three-dimensional model will be described with reference to FIG.
The three-dimensional model 25a is generated using a certain stereo pair image. The subject is an object on a quadrangular prism, and in the three-dimensional model 25a, a surface M1 that is an upper surface and a surface M2 that is a side surface are generated among the surfaces constituting the subject.

The three-dimensional model 25b is generated using another stereo pair image. In the three-dimensional model 25b, a surface M1 and a surface M3 which are side surfaces among the surfaces constituting the subject are generated.
Here, assuming that both three-dimensional models share common points K1 to K4, the three-dimensional model 25c is obtained by matching the spatial coordinates of both three-dimensional models using the common points.

In the three-dimensional model 25c, among the surfaces constituting the subject, M1 to M3 are formed, and a surface M3 that is not generated due to a blind spot in the three-dimensional model 25a is formed.
Furthermore, by connecting a three-dimensional model based on stereo pair images taken from other directions (periphery, upper part, etc.), the omnidirectional object can be configured.

As described above, the three-dimensional model connection unit 68 includes the three-dimensional model 25a generated from one stereo pair image (first still image and second still image) and another stereo pair image (third still image). A coordinate conversion unit is provided for performing coordinate conversion so that the three-dimensional coordinate axes of the three-dimensional model 25b generated from the image and the fourth still image match.
Further, one stereo pair image and another stereo pair image may be obtained by tracking the same orientation point (that is, the same orientation point being tracked from the first still image to the fourth still image). ) Or regenerated (for example, after acquiring the first still image and the second still image, and then regenerating the orientation point to acquire the third still image and the fourth still image) ) In the latter case, coordinate conversion is performed using a common point.

Also, when there are a plurality of subjects in the moving image data, they can be linked.
For example, as shown in FIG. 11A, consider a case where subjects 31, 32 are photographed in frame A, subjects 32, 33 are photographed in frame B, and subjects 33, 34 are photographed in frame C. .
Coordinate systems 31a, 32a, 33a, and 34a are set for the subjects 31, 32, 33, and 34, respectively.

The positional relationship between the coordinate system 31a and the coordinate system 32a is known from the frame A, and the positional relationship between the coordinate system 33a and the coordinate system 34a is known from the frame C.
Since the positional relationship between the frame 32a and the frame 33a is known from the frame B, all the positional relationships between the coordinate system 31a to the coordinate system 34a are known.
Therefore, as shown in FIG. 11B, a three-dimensional model of each subject can be generated including the relative positional relationship between the subjects 31 to 34.
Thereby, for example, it is possible to take a moving image of a cityscape where buildings are built from cars, and use this to convert the cityscape into computer graphics.

Next, a procedure for generating a three-dimensional model will be described with reference to the flowchart of FIG.
The following processing is performed by a CPU (Central Processing Unit) included in the three-dimensional information processing apparatus 1 according to the three-dimensional information processing program.
Note that the i-th frame is represented as F (i) or the like.

First, the moving image data processing unit 62 acquires moving image data. Then, the counter i is set to 1, and F (i) = F (1), that is, initial processing is performed on the first frame (step 5).
The counter i is i shown in FIG. 5, and the moving image data processing unit 62 selects a frame constituting the stereo pair image from the i-th to i + β-th frames.
In the initial processing, the control point setting unit 63 sets a control point by generating a mesh on the subject photographed at F (1), and further receives a specification of the reference point from the user by the reference point specifying unit 64. Is done.

When the initial process is completed as described above, the orientation point tracking unit 65 starts tracking the orientation points.
The orientation point tracking unit 65 first sets the counter k to 1 (stereo pair image 10). The counter k is k shown in FIG. 5 and is a parameter for counting frames for tracking the orientation points.
After setting the counter k, the orientation point tracking unit 65 acquires an orientation point at F (i + k) using the tracking technique (step 15).

Next, the orientation point tracking unit 65 determines whether or not the orientation point has disappeared by 30% or more at F (i + k) (step 20).
If 30% or more has not disappeared (step 20; N), the orientation point tracking unit 65 further determines whether or not there are less than four reference points (step 25).
When the reference points disappear and become less than 4 (step 25; Y), the reference points are designated again (step 30). This designation is performed when the reference point designation unit 64 receives designation from the user.

When there are four or more reference points (step 25; N), or after specifying a reference point (step 30), the orientation point tracking unit 65 sets the counter j to 1 (step 35).
Then, the orientation point tracking unit 65 determines whether the vertical parallax between F (i + k−j) and F (i + k) is less than α (step 40).
When the vertical parallax is less than α (step 40; Y), the orientation point tracking unit 65 adopts these two frames as a pair of stereo pair images, and orientation points of F (i + k−j) and F (i + k). The information is held in the orientation point information holding unit 66 (step 85).

If the vertical parallax is greater than or equal to α (step 40; N), the orientation point tracking unit 65 does not adopt the two combinations as a stereo pair image, and further determines whether j is smaller than k ( Step 45).
When j is smaller than k (step 45; Y), the orientation point tracking unit 65 adds 1 to j (step 50) and returns to step 40.

When j is greater than or equal to k (when j is equal to 1 as a result of incrementing 1) (step 45; N), the orientation point tracking unit 65 further determines whether k is smaller than β (step 55). ).
When k is smaller than β (step 55; Y), the orientation point tracking unit 65 adds 1 to k and returns to step 15.

When k is equal to or larger than β (when it is equal to β as a result of incrementing 1 to k) (step 55; N), the orientation point tracking unit 65 adds β to i.
And the control point tracking part 65 confirms the magnitude relationship between i and the maximum number G of frames, and when i is smaller than G (step 65; Y), it returns to step 10 and i becomes G or more. (Step 65; N), the control process for obtaining the control point information is terminated.

  Then, the 3D model generation unit 67 generates a 3D model for each stereo pair image using the orientation point information held in the orientation point information holding unit 66 (step 70), and the 3D model connection unit 68 generates The obtained three-dimensional models are connected and output (step 75).

  On the other hand, as tracking progresses, the disappearance of ground control points increases. For this reason, if the orientation point disappears by 30% or more (step 20; N), it is necessary to set the orientation point again, so a rectangular area is set on the subject by F (i + k) and a mesh is generated (step) 100).

And the control point tracking part 65 uses the thing which overlaps with the control point before resetting among the resetting control points as a reference point.
Next, the orientation point tracking unit 65 checks whether or not the number of reference points is less than 4 (step 105), and if it is less than 4 (step 105; Y), the reference point specifying unit 64 A reference point designation from the user is accepted (step 110).

  When there are four or more reference points (step 105; N), or after receiving the designation of the reference point, the orientation point tracking unit 65 sets i to i + k (step 115), and proceeds to step 60.

  FIG. 13 is a block diagram illustrating a hardware configuration of the three-dimensional information processing apparatus 1. The three-dimensional information processing apparatus 1 can be configured using, for example, a personal computer.

The three-dimensional information processing apparatus 1 includes an input unit 82, an output unit 83, a communication control unit 84, a storage unit 85, a storage medium drive unit 88, and an input / output interface (I / F) 89 via a bus line 77. Etc. are connected. The bus line 77 is a transmission line for transmitting signals between the CPU 78 and other devices.
The control unit 76 includes a CPU 78, a ROM (Read Only Memory) 80, a RAM (Random Access Memory) 81, and the like.

  The CPU 78 executes a three-dimensional information processing program, and performs a series of three-dimensional information processing such as acquisition of still images from moving image data, setting of control points, selection of stereo pair images, photogrammetry, and generation and connection to a three-dimensional model. Do.

  The ROM 80 is a read-only memory that stores various programs, data, parameters, and the like for the CPU 78 to perform various calculations and controls. The CPU 78 can read programs, data, parameters, and the like from the ROM 80.

  The RAM 81 provides a random access memory used as a working memory to the CPU 78. The CPU 78 can write and erase programs and data in the RAM 81. In the present embodiment, the RAM 81 is used as a temporary storage device when the CPU 78 executes a three-dimensional information processing program.

The input unit 82 includes an input device such as a keyboard, a mouse, and a joystick.
The keyboard is a device for inputting information such as characters and numbers to the three-dimensional information processing apparatus 1.
The keyboard includes keys for inputting kana and English characters, numeric keys for inputting numbers, various function keys, cursor keys, and other keys.

In addition to inputting text data to the three-dimensional information processing apparatus 1, various commands can be input using the keyboard. The measured value of the reference point can be input from the keyboard.
A mouse is a pointing device. When the three-dimensional information processing apparatus 1 can be operated using a GUI (Graphical User Interface), a reference point can be specified by clicking a ground point or other point on the frame displayed on the display device. .

The output unit 83 includes, for example, a display device and a printing device.
The display device is configured by a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or a plasma display, and is a device for presenting image information, character information, and the like on a screen.
The display device can display a frame acquired from moving image data or display a generated three-dimensional model.

  The printing apparatus is an apparatus that prints information on a print medium such as paper. The printing apparatus includes various printer apparatuses such as an ink jet printer, a laser printer, a thermal transfer printer, and a dot printer. Frames, orientation point information, 3D models, etc. can be printed.

The communication control unit 84 is a device for connecting to a network such as the Internet via a communication line, and includes a modem, a terminal adapter, and other devices.
When moving image data is acquired via a network, it is received from the server device via the communication control unit.

  The storage unit 85 includes a readable / writable storage medium and a drive device for reading / writing programs and data from / to the storage medium. A hard disk is mainly used as the storage medium, but it can also be configured by other readable / writable storage media such as a magneto-optical disk, a magnetic disk, and a semiconductor memory.

In the storage unit 85, a program storage unit 86 and a data storage unit 87 are formed.
The program storage unit 86 stores a three-dimensional information processing program, an OS (Operating System) that realizes basic functions for operating the three-dimensional information processing apparatus 1, and the like.
The CPU 78 reads out and executes the three-dimensional information processing program from the program storage unit 86, whereby the components as shown in FIG.
The data storage unit 85 stores moving image data and a three-dimensional model generated from the moving image data, and also forms a control point information holding unit 66 and stores control point information.

The storage medium driving unit 88 is a driving device for driving a removable storage medium to read / write data. Examples of the removable storage medium include a magneto-optical disk, a magnetic disk, a magnetic tape, a semiconductor memory, a paper tape punched with data, and a CD-ROM. Note that CD-ROMs and paper tapes can only be read.
By driving the storage medium drive unit 88, moving image data stored in the storage medium can be read, and the generated three-dimensional model can be written in the storage medium.

  The input / output interface 89 is constituted by, for example, a serial interface or another standard interface. The input / output interface 89 is an interface for connecting an external device to the three-dimensional information processing apparatus 1. In this embodiment, it can be used to connect to a digital video camera and read moving image data from the camera.

In the above, the three-dimensional information processing apparatus 1 is configured using a personal computer or the like. However, the present invention is not limited to this, and can be incorporated into, for example, a digital video camera.
The digital video camera includes a CPU, a storage medium, a liquid screen, input keys, and the like, and a three-dimensional information processing program can be used by being incorporated in the digital video camera.
The orientation point may be tracked in the direction in which time passes in the moving image data, or may be tracked in the direction of going back in time.

In the present embodiment described above, the following effects can be obtained.
(1) A subject can be photographed with a video camera, and a three-dimensional model of the subject can be generated from the moving image data.
(2) An orientation point can be automatically set for a subject using an image processing technique.
(3) By tracking a control point with moving image data using tracking technology, a stereo pair image having a corresponding control point can be acquired from the moving image data.
(4) A plurality of stereo pair images can be acquired from moving image data to generate a plurality of three-dimensional models, and these can be connected using a common point.
(5) A new orientation point can be automatically set when the orientation point on the subject disappears due to movement.
(6) When resetting the control points, the continuity of the spatial information can be maintained before and after the automatic setting by setting the control points so that at least three of the control points before automatic setting overlap.

It is a figure for demonstrating the outline | summary of the procedure which produces | generates a three-dimensional model. It is the block diagram which showed the functional structure of the three-dimensional information processing apparatus. It is a figure for demonstrating the procedure in which an orientation point setting part sets an orientation point. It is a figure for demonstrating the tracking method of an orientation point. It is a figure for demonstrating the method to select the flame | frame which comprises a stereo pair image. It is a figure for demonstrating the case where an orientation point is reset. It is a figure for demonstrating the other case where a control point is reset. It is the table | surface which showed an example of the control point information which the control point tracking part extracted. It is a flowchart for demonstrating the procedure of a photogrammetry process. It is a figure for demonstrating the connection of a three-dimensional model. It is a figure for demonstrating the case where a some 3D model is connected. It is a flowchart for demonstrating the production | generation procedure of a three-dimensional model. It is the block diagram which showed the hardware-like structure of the three-dimensional information processing apparatus. It is a figure for demonstrating photogrammetry.

Explanation of symbols

1 3D information processing device 2 Subject 3 Video camera 6 3D model

Claims (7)

Moving image data acquisition means for acquiring moving image data of a subject while changing the shooting position;
First still image acquisition means for acquiring, from the acquired moving image data, a first still image in which an orientation point is set for the subject;
Corresponding image acquisition means for acquiring a second still image having a control point corresponding to the set control point by tracking the movement of the set control point using the moving image data;
From the two-dimensional coordinate value of the orientation point on the first still image and the two-dimensional coordinate value of the orientation point on the second still image, the three-dimensional coordinate value of the orientation point on the subject is calculated. Three-dimensional coordinate value calculating means;
A three-dimensional information processing apparatus comprising: Comprising a control point setting means for acquiring a still image from moving image data and setting a control point on the subject in the still image;
The first still image acquisition unit acquires a still image in which the orientation point is set or a still image in which the orientation point is tracked from the still image using the moving image data as the first still image. The three-dimensional information processing apparatus according to claim 1. Criteria for accepting designation of reference points whose measured values of at least two plane positions and measured values of heights of at least three points are known among the orientation points of the first still image or the second still image. Point acceptance means;
An actual value input means for receiving an input of an actual value of the received reference point;
Comprising
3. The three-dimensional information processing apparatus according to claim 1, wherein the three-dimensional coordinate value calculation unit calculates an actual size three-dimensional coordinate value by using an actual measurement value of the reference point. Still image acquisition means for acquiring two still images having corresponding control points from the moving image data by tracking the control points set on the subject;
Evaluation means for evaluating the accuracy of the three-dimensional coordinate value of the subject calculated from the obtained two still images using the orientation point;
Comprising
The said 1st still image acquisition means and the said corresponding | compatible image acquisition means acquire the still image evaluated as having a predetermined precision by the said evaluation means, The claim 2, The claim 2, or Claim characterized by the above-mentioned. 3. The three-dimensional information processing apparatus according to 3. Comprising a control point resetting means for newly setting a control point in the second still image so as to overlap with at least three control points of the second still image,
The corresponding image acquisition means acquires the third still image and the fourth still image having the resetting point by tracking the movement of the resetting point with the moving image data,
The three-dimensional coordinate value calculation means calculates a standard on the subject from the two-dimensional coordinate value of the orientation point on the third still image and the two-dimensional coordinate value of the orientation point on the fourth still image. Calculate the three-dimensional coordinate value of the fixed point,
The three-dimensional coordinate axis calculated from the first still image and the second still image matches the three-dimensional coordinate axis calculated from the third still image and the fourth still image. 5. The coordinate conversion unit according to claim 1, further comprising coordinate conversion means for performing coordinate conversion using the coordinate values of the orientation points generated in duplicate. 6. 3D information processing device. In a computer comprising moving image data acquisition means, first still image acquisition means, corresponding image acquisition means, and three-dimensional coordinate value calculation means,
A moving image data acquisition step of acquiring moving image data obtained by shooting the subject while changing the shooting position by the moving image data acquisition means;
A first still image acquisition step of acquiring, from the acquired moving image data, a first still image in which an orientation point is set for the subject by the first still image acquisition means;
Corresponding image for acquiring a second still image having a control point corresponding to the set control point by tracking the movement of the set control point using the moving image data by the corresponding image acquisition means. An acquisition step;
From the two-dimensional coordinate value of the orientation point on the first still image and the two-dimensional coordinate value of the orientation point on the second still image by the three-dimensional coordinate value calculation means, a standard on the subject is obtained. A three-dimensional coordinate value calculating step for calculating a three-dimensional coordinate value of the fixed point;
A three-dimensional information processing method characterized by comprising: A video data acquisition function that acquires video data of a subject while changing the shooting position;
A first still image acquisition function for acquiring, from the acquired moving image data, a first still image in which an orientation point is set for the subject;
A corresponding image acquisition function for acquiring a second still image having a control point corresponding to the set control point by tracking the movement of the set control point using the moving image data;
From the two-dimensional coordinate value of the orientation point on the first still image and the two-dimensional coordinate value of the orientation point on the second still image, the three-dimensional coordinate value of the orientation point on the subject is calculated. A three-dimensional coordinate value calculation function;
A three-dimensional information processing program that implements

Images