JP2005165468A - Three-dimensional image display apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional image display apparatus that can integrate and visualize 3D measurement data obtained from a stereo image with an image of a measuring object with three-dimensional texture. <P>SOLUTION: The apparatus includes: an orientation section 24 for finding relationship as to corresponding points in a stereo image of a measuring object 1 based on the position and the tilt at which the stereo image is photographed; a three-dimensional coordinate data section 31 for obtaining three-dimensional coordinate data on the corresponding points of the measuring object 1 based on the relationship as to the corresponding points found by the orientation section 24; a model forming section 32 for forming a model of the measuring object 1 based on the three-dimensional coordinate data on the corresponding points; an image correlating section 34 for correlating the stereo image of the measuring object stored in the stereo image data storage section 12 and the model formed by the model forming section 32, using the relationship as to the corresponding points found by the orientation section 24; and a model display section 35 for displaying an image of the measuring object 1 with three-dimensional texture, using the stereo image correlated with the model by the image correlating section 34. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  This invention relates to 3D data (three-dimensional shape data) measured with respect to a measurement object and a stereo image of the measurement object, and enables the measurement data and image of the measurement object to be displayed simultaneously. The present invention relates to an image display apparatus and method.

  As a method of acquiring 3D data of a construction object and a production object, a method of acquiring by a three-dimensional position measuring device (total station) as described in Patent Document 1 and Patent Document 2 are described. There is a stereo image measurement method that acquires 3D data by photographing a stereo image using a target object and a comparative calibration body and performing stereo measurement. In the method using the three-dimensional position measuring device, the accuracy of the obtained 3D coordinates is good, and therefore, it is used for measuring the reference point position when pasting an image. In particular, in a recent total station, a relatively large number of three-dimensional coordinates (for example, about several tens of points for a measurement object) have been obtained by driving a motor. In the stereo image measurement method, three-dimensional coordinates of about several thousand to several tens of thousands of points can be obtained relatively easily by performing an operation called orientation when pasting 3D data and an image.

JP 2002-352224 A FIG. Japanese Patent Laid-Open No. 2003-65737 FIG. 1 and FIG.

  However, the 3D data acquired by the 3D position measuring device is basically composed of 3D coordinate data including distance data. Therefore, it is difficult to associate the 3D data acquired by the three-dimensional position measurement device with the on-site situation, and any position of the measurement object is measured when performing the operation of associating the 3D data with the image information of the measurement object. It turned out that there was a problem that it would be difficult.

  On the other hand, in the stereo image measurement method, since 3D measurement is performed using a stereo image, 3D data and the stereo image can be compared by displaying the image in stereo. However, in order to compare 3D data and stereo images, a stereoscopic monitor and deflection glasses are required, and there are some people who cannot and cannot easily confirm stereoscopic viewing because there are people who can and cannot do stereoscopic viewing. . Further, in the stereo measurement of the measurement object and the work of associating the 3D data with the stereo image, it is necessary to perform the orientation work by associating the image information with the 3D data or between the images. This orientation work has a problem that individual differences among workers are large and cannot be easily performed with high accuracy.

  The present invention solves the above-described problems, and an object of the present invention is to provide a three-dimensional image display apparatus and method that can visualize 3D measurement data acquired from a stereo image by integrating it with an image with a three-dimensional texture of a measurement object. And

  As shown in FIG. 1, the three-dimensional image display device of the present invention that achieves the above object is based on a stereo image data storage unit 12 that stores a stereo image of a measurement object 1, and a shooting position and inclination related to the stereo image. The orientation unit 24 for obtaining the corresponding point relationship of the stereo image, the 3D coordinate data unit 31 for obtaining the 3D coordinate data of the corresponding point of the measuring object 1 from the corresponding point relationship obtained by the orientation unit 24, and the correspondence A model forming unit 32 that forms a model of the measurement object 1 from the three-dimensional coordinate data of the points, a stereo image of the measurement object stored in the stereo image data storage unit 12, and a model formed by the model formation unit 32. Measurement using the image correspondence unit 34 associated with the corresponding point relationship obtained by the orientation unit 24 and the stereo image associated with the model by the image correspondence unit 34 And a model display section 35 for displaying a stereoscopic texture image of object (1).

  According to such a configuration, the orientation unit 24 performs relative orientation on the stereo image of the measuring object 1 stored in the stereo image data storage unit 12 to obtain the corresponding point relationship of the stereo image. Next, the three-dimensional coordinate data unit 31 obtains the three-dimensional coordinate data of the corresponding point of the measurement object 1 using the corresponding point relationship obtained by the orientation unit 24, and the model forming unit 32 determines the measurement object 1. Form a model. The image correspondence unit 34 associates the stereo image of the measurement object with the model using the corresponding point relationship obtained by the orientation unit 24. The model display unit 35 displays an image with a three-dimensional texture of the measurement object 1 using a stereo image associated with the model. The three-dimensional textured image expresses the measurement object 1 as a two-dimensional image, but expresses a pseudo three-dimensional expression by expressing the surface unevenness using a shadow.

  Preferably, in the three-dimensional image display device of the present invention, a posture instruction unit 36 for instructing the posture of the model of the measurement object 1, and coordinates for performing coordinate conversion of the corresponding points in accordance with the posture instruction for the model The model display unit 35 may be configured to display an image with a three-dimensional texture of the measurement object 1 according to the posture instructed by the posture instruction unit 36. When the orientation of the model of the measuring object 1 is instructed by the orientation instructing unit 36, the coordinate conversion unit 37 performs coordinate conversion and displays an image with a three-dimensional texture of the measuring object 1 according to the orientation instructed by the model display unit 35. Therefore, an image with a three-dimensional texture in an arbitrary direction of the measurement object 1 is obtained.

  Preferably, in the three-dimensional image display device of the present invention, the image corresponding unit 34 further covers the stereo image with a unit image plane formed by a plurality of corresponding points, and the model and stereo are displayed using the unit image plane. It is preferable to associate the images. By covering the stereo image with the unit image surface, it is possible to easily associate the stereo image of the measurement object with the model. As the unit image plane, a triangular plane using three adjacent corresponding points or a rectangular plane using four adjacent corresponding points can be selected.

  Preferably, the three-dimensional image display device of the present invention further includes a corresponding point instruction unit 22 for indicating a corresponding point used in the orientation unit 24, and the corresponding point instruction unit 22 is provided on the display device for displaying the stereo image. The displayed measurement object 1 may be configured to show feature points in the vicinity of the designated position. The feature points near the designated position include, for example, the center position, the center of gravity position, and the corner position of the measurement object 1. When the corresponding point is instructed by the corresponding point instructing unit 22, even if the operator does not strictly instruct the feature point, the operator is drawn into the characteristic point originally intended by the operator, so that the orientation work by the orientation unit can be easily performed.

  As shown in FIG. 2, the 3D image display method of the present invention that achieves the above object is a step of obtaining corresponding point relationships of stereo images based on the shooting position and inclination of the stereo image of the measuring object 1 by the orientation unit 24, as shown in FIG. (S30), a step of obtaining the three-dimensional coordinate data of the corresponding point of the measuring object 1 from the corresponding point relationship obtained by the orientation unit 24 by the three-dimensional coordinate data unit 31 (S40, S50), and a model forming unit The step of forming a model of the measuring object 1 from the three-dimensional coordinate data of the corresponding point (S60) by 32, and the stereo image of the measuring object 1 and the model formed by the image corresponding unit 34, The object to be measured using the stereo image associated with the model by the step of associating using the corresponding point relationship obtained in 24 (S72) and the model display unit 35. Stereoscopic texture image is intended to be executed by a computer and a step (S80) of displaying.

  According to the three-dimensional image display apparatus of the present invention, the stereo image of the measurement object 1 by the image corresponding unit 34 and the model formed by the model forming unit 32 are used using the corresponding point relationship obtained by the orientation unit 24. Since the correspondence and model display unit 35 is configured to display an image with a three-dimensional texture of the measurement object 1 using a stereo image associated with the model, 3D data measured in stereo is also stereoscopic. The 3D data and the image of the measuring object 1 can be easily confirmed without performing stereoscopic viewing using a viewing device.

  The present invention will be described below with reference to the drawings. The three-dimensional image display device of the present invention calculates the three-dimensional shape of the measuring object 1 from two or more images taken at least as a unit of two left and right images constituting a stereo image. The texture (texture) that expresses the three-dimensional effect is pasted on the two-dimensional image of the measurement object 1 using the 3D data obtained by 3D measurement or the entire measurement object 1. Here, the texture refers to a pattern attached to the surface of a graphic or a drawing for expressing the texture in graphics or the like.

  FIG. 1 is a block diagram showing the overall configuration of a first embodiment of the present invention. The present invention includes a stereo image data storage unit 12, a measurement object position data storage unit 16, a corresponding point instruction unit 22, an orientation unit 24, a display image forming unit 30, and a display device 40. For example, a computer and a display device are used. It is configured. The measurement object 1 is a tangible object that is a construction object or a production object, and corresponds to, for example, various works such as buildings, people, and landscapes. The image capturing device 10 is a device that combines, for example, a stereo camera or a general-purpose digital camera and a device that compensates lens aberration for the left and right images of the measurement object 1 captured by the digital camera. The three-dimensional position measuring device 14 is a device that automatically measures a target position to be surveyed using a laser or the like, and includes a total station, a three-dimensional scanner, and the like. Here, the three-dimensional scanner emits laser light to the measurement object 1, receives the reflected light with a CCD camera in the machine, obtains distance data from the received light by the principle of triangulation, and performs three-dimensional scanning. It is used for various purposes such as industrial design, creating a database of 3D images, and 3D CG in video production. By scanning the measurement object 1 with laser light from top to bottom, it is possible to input a distance image of, for example, 640 × 480 points in the entire irradiation image in one scan.

  The stereo image data storage unit 12 stores a stereo image of the measurement target 1 and stores, for example, left and right images of the measurement target 1 captured by the image capturing device 10. The measurement object position data storage unit 16 stores position data of the measurement object 1 serving as three reference points required for absolute orientation. For example, the reference point measured by the three-dimensional position measurement device 14 The position data is stored. Note that the object position data storage unit 16 does not need to be provided in applications where relative orientation using a model coordinate system is sufficient without performing absolute orientation.

  The corresponding point instruction unit 22 is for instructing a corresponding point used in the orientation unit 24, and is preferably configured to indicate a feature point in the vicinity of the indicated position with respect to the measurement object 1 displayed on the display device 40. The feature points near the designated position include, for example, the center position, the center of gravity position, and the corner position of the measurement object 1. When the corresponding point is instructed by the corresponding point instructing unit 22, the operator is drawn into the characteristic point originally intended by the operator even if the operator does not strictly instruct the characteristic point, so that the orientation work by the orientation unit 24 can be easily performed. . The orientation unit 24 obtains the corresponding point relationship of the stereo image based on the shooting position and the inclination regarding the stereo image. Details of the corresponding point instruction unit 22 and the orientation unit 24 will be described later.

  The display image forming unit 30 includes a three-dimensional coordinate data unit 31, a model forming unit 32, a model storage unit 33, an image corresponding unit 34, a model display unit 35, a posture instruction unit 36, and a coordinate conversion unit 37. The three-dimensional coordinate data unit 31 obtains the three-dimensional coordinate data of the corresponding point of the measurement object 1 from the corresponding point relationship obtained by the orientation unit 24. The model forming unit 32 forms a model of the measurement object 1 from the three-dimensional coordinate data of the corresponding points. The model storage unit 33 stores a model of the measurement object 1 formed by the model forming unit 32. The image correspondence unit 34 associates the stereo image of the measurement object stored in the stereo image data storage unit 12 with the model formed by the model formation unit 32 using the corresponding point relationship obtained by the orientation unit 24. The model display unit 35 displays the measurement target 1 as a stereoscopic two-dimensional image using a stereoscopic textured image such as a bird's-eye image using the stereo image associated with the model by the image corresponding unit 34. Display on device 40.

  The posture instruction unit 36 instructs the posture of the model of the measurement object 1. For example, the operator operates a cursor input device such as a mouse to indicate the posture of the measurement object 1 displayed on the display device 40. To do. The coordinate conversion unit 37 performs coordinate conversion of corresponding points in accordance with an orientation instruction for the model. The model display unit 35 displays an image with a three-dimensional texture of the measurement object 1 according to the posture instructed by the posture instruction unit 36. The display device 40 is an image display device such as a liquid crystal display device or a CRT.

  Next, the operation of the three-dimensional image display device of the present invention configured as described above will be described. FIG. 2 is a flowchart for explaining the operation of the three-dimensional image display apparatus shown in FIG. First, two or more images of the measurement object 1 photographed using the image photographing device 10 such as a digital camera are registered in the stereo image data storage unit 12 (S10). Next, among the images registered in the stereo image data storage unit 12, a set of left and right images forming a stereo pair is set (S20). However, it is not necessary to set the stereo pair for all the images registered in the stereo image data storage unit 12 in S20. That is, you may set as a stereo pair which comprises a stereo image at any time in order from the stereo image which wants to measure, and the stereo image which wants to paste a three-dimensional texture.

  Next, it is determined whether or not the three-dimensional coordinates of the measuring object 1 obtained using the three-dimensional position measuring device 14 or the 3D scanner exist (S22). If Yes in S22, position data using the three-dimensional coordinates of the measurement object 1 is read into the measurement object position data storage unit 16 (S24). If No in S22, the process can be performed in the relative coordinate system, so S24 may be skipped. The reading of the position data using the three-dimensional coordinates of the measurement object 1 with respect to the measurement object position data storage unit 16 is not limited to after S20. For example, processing is performed in a relative coordinate system until S80, After S80, the three-dimensional coordinates may be read to form an absolute coordinate system. The three-dimensional coordinates as reference points can be calculated and measured as an absolute coordinate system if there are at least three or more points in all image areas where measurement or stereoscopic texture is to be pasted.

  Next, the orientation unit 24 performs orientation work, performs relative orientation of the stereo image of the measuring object 1 stored in the stereo image data storage unit 12, and obtains the corresponding point relationship of the stereo image (S30). This orientation work has three modes: manual, semi-automatic, and fully automatic. Details will be described later. Here, the orientation operation is an operation in which corresponding points (same points) of two or more images are indicated on each image by the corresponding point instruction unit 22 with a mouse cursor or the like, and the image coordinates are read. Normally, 6 or more corresponding points are required for each image. If the reference point coordinates are stored in the measurement object position data storage unit 16 in S24, the absolute orientation is executed by associating the reference point coordinates with the image.

  The orientation calculation processing is performed by the orientation unit 24 using the coordinates of the corresponding points obtained by the orientation work (S40). By the orientation calculation process, the position and tilt of the photographed camera, the position of the corresponding point, and the measurement accuracy can be obtained (see FIG. 3). If there is a reference point coordinate, it is read from the measurement object position data storage unit 16 and an absolute orientation calculation process is also performed. If there are no reference point coordinates, the calculation is performed using relative (model) coordinates. The orientation calculation processing is performed by mutual orientation for associating stereo models, and orientation for all images is performed by bundle adjustment. Details of mutual orientation will be described later.

  FIG. 3 is a screen diagram illustrating an example of the orientation result of the orientation calculation process. The orientation result screen diagram 100 includes a result list screen 102, a pass point screen 104, an orientation point screen 106, a calculated coordinate screen 108, an imaging situation and ground resolution screen 110. The shooting situation and ground resolution screen 110 is provided with a shooting situation of stereo images and a ground resolution display surface 120, and a camera position and tilt display surface 130. On the stereo image shooting status and ground resolution display surface 120, a pair name column 121, a baseline length B column 122, a shooting distance H column 123, a B / H ratio column 124, which display two pieces of image data forming a stereo pair, A plane resolution column 125 and a depth resolution column 126 are provided. The camera position and tilt display surface 130 includes an image name column 131, an Xo column 132 indicating the three-dimensional coordinates of the image origin, a Yo column 133, a Zo column 134, and a ω column indicating the tilt angle of the image from the reference coordinate system. 135, φ column 136, and κ column 137 are provided.

  Returning to FIG. 2, stereo measurement is performed in the three-dimensional coordinate data unit 31 to obtain the three-dimensional coordinates of the measurement object 1 (S 50). Alternatively, even when stereo measurement is not performed, when the three-dimensional coordinates of the corresponding points of the stereo image are read in advance into the measurement object position data storage unit 16, the three-dimensional coordinate data unit 31 stores the measurement object position data. The three-dimensional coordinates of the corresponding points are read from the unit 16. For stereo measurement, there are various modes of manual measurement, semi-automatic measurement, and automatic measurement as disclosed in, for example, Japanese Patent Laid-Open No. 2003-284098 proposed by the present inventor. Therefore, in stereo measurement, a stereo (right and left) image that can be viewed stereoscopically is created and displayed according to the orientation calculation processing result of S40, and the three-dimensional coordinates of the corresponding points are obtained using the above-described mode. When performing in the automatic measurement mode, automatic measurement is performed by specifying a measurement region for automatic measurement. In manual and semi-automatic measurement, while observing the left and right stereo screens, the corresponding points on the left and right images are measured semi-automatically or while the points corresponding to the manual are determined with the mouse.

FIG. 4 is a diagram illustrating an example of stereo measurement performed on a stereo image. Here, as an example of the stereo image 150, a ruins of a temple is the measurement object 1, and the left image 150L and the right image 150R are a stereo pair. Here, the left image 150L is a reference image, and the coordinates of the feature points of the left image 150L are (X ₁ , Y ₁ ). Then, the three-dimensional coordinate data unit 31 treats the right image 150R as a search image, and the coordinates of the corresponding point of the right image 150R corresponding to the feature point of the left image 150L are indicated by (X ₂ , Y ₂ ).

  FIG. 5 is a diagram illustrating another example of stereo measurement performed on a stereo image. Here, a water bottle transport relief formed on the wall surface of the temple ruins is adopted as the measurement object 1. As the stereo image 160 of the measurement object 1, the left image 160L and the right image 160R of the water bottle transport relief are in a stereo pair. Here, the three-dimensional coordinate data unit 31 handles the left image 160L as a reference image and the right image 150R as a search image. Then, an image region 162L near the feature point of the left image 160L and an image region 162R near the corresponding point of the right image 160R are extracted and displayed in an enlarged manner in FIG.

  FIG. 6 is an enlarged view of the image area 162L near the feature point and the image area 162R near the corresponding point. When the operator designates a corresponding point of the left image 150L (reference image), the corresponding point instruction unit 22 is drawn into a feature point near the designated position of the right image 150R (search image). Here, the corner position of the water bottle transport relief is shown as a feature point near the indicated position.

  Returning to FIG. 2, the model forming unit 32 creates a model of the measuring object 1 from the obtained three-dimensional coordinates or the read three-dimensional coordinates (S60). FIG. 7 is a diagram illustrating an example of a model of the measurement object 1 and illustrates a wire frame surface. Here, the water bottle carrying relief formed on the wall surface of the temple ruins is used as the measurement object 1 as the stereo image 170. The left image 170L and the right image 170R of the water bottle transport relief are in a stereo pair. Wire frame surfaces 174L and 174R are formed inside the left and right region designation frame lines 172L and 172R. As the unit image surfaces constituting the wire frame surfaces 174L and 174R, for example, a triangular surface using three adjacent corresponding points or a rectangular surface using four adjacent corresponding points can be selected. At this time, the orientation of the surface created first is set as the default value for the orientation of the model of the measurement object 1.

  Returning to FIG. 2, the image corresponding unit 34 designates a surface to be texture-mapped (S70). Texture mapping refers to pasting a texture expressing a three-dimensional effect on the two-dimensional image of the measurement object 1. The process of S70 may also be the measurement area designation performed in S50 when automatic stereo measurement is performed. Next, the image corresponding unit 34 texture maps the stereo image of the measurement target 1 and the model formed by the model forming unit 32 using the corresponding point relationship obtained by the orientation unit 24 (S72). Details of the texture mapping will be described later.

  Next, the model display unit 35 displays the texture-mapped image on the model screen (S80). The model screen is a two-dimensional image of the measurement object 1 representing a perspective state, such as an image with a three-dimensional texture formed from a stereo image or an image from which the texture is removed. The display in S80 may display not only a texture-mapped image but also a wire frame image, a three-dimensional point group (for example, a point image acquired by a 3D scanner), a camera shooting position, and a reference point position. Further, by displaying the objects displayed on the model screen at the same time or by switching them, confirmation of the stereo image measurement result and the captured image of the measurement object 1 becomes easy.

  FIG. 8 is a diagram in which a texture expressing a three-dimensional effect is expressed by a wire frame. On the wire frame screen 180, for example, an area designation frame line 182 of the measurement object 1 and a wire frame 184 of the measurement object 1 are displayed. The wire frame 184 is affixed to the two-dimensional image of the measuring object 1 that represents a perspective state.

  FIG. 9 is a display example of texture mapping, and shows a case where the measurement object 1 is displayed as an image with a three-dimensional texture. On the texture mapping screen 190, for example, an area designation frame line 192 of the measurement object 1 and a bird's-eye view image 194 of the measurement object 1 on which a texture having a three-dimensional effect is mapped are displayed by the model display unit 35. The bird's-eye view image 194 of the measurement object 1 is displayed as a two-dimensional image of the measurement object 1 representing a perspective state.

  Returning to FIG. 2, the operator instructs the display direction of the measurement object 1 on the wire frame screen 180 or the texture mapping screen 190 by the posture instruction unit 36 using a mouse, a keyboard, or the like (S90). Then, the coordinate conversion unit 37 converts the display direction of the measurement object 1 displayed on the display device 40 into the direction instructed by the posture instruction unit 36 and displays it on the wire frame screen 180 and the texture mapping screen 190. (S92). The operator determines whether the display direction of the other measurement object 1 exists (S94). If Yes, the process returns to S90, and if No, the process ends (S96). By displaying the measurement result and the measurement object 1 on the display device 40 by changing the viewpoint position from any angle by the function that can arbitrarily specify the display direction of the measurement object 1 such as S90 and S92, the operator can display the measurement result and the measurement object 1 on the display device 40. It is possible to visually confirm the measuring object 1.

[Mutual orientation]
Next, the relative orientation performed by the orientation unit 24 will be described. FIG. 10 is an explanatory diagram of a model coordinate system XYZ and a camera coordinate system xyz in a stereo image. The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2.

この場合左側のカメラのＸ軸の回転角ω_１は０なので、考慮する必要ない。 First, parameters required to determine the positions of the left and right cameras are obtained by the following coplanar conditional expression (1).

In this case, since the rotation angle ω ₁ of the X axis of the left camera is 0, it is not necessary to consider.

ここで、モデル座標系ＸＹＺとカメラ座標系ｘｙｚの間には、次に示すような座標変換の関係式（３）、（４）が成り立つ。

Under the above conditions, the coplanar conditional expression (1) is transformed as shown in Expression (2), and each parameter can be obtained by solving Expression (2).

Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (3) and (4) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(I) The initial approximate value is normally 0.
(Ii) The coplanar conditional expression (2) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained from the expressions (3) and (4), and an observation equation is established.
(Iii) Applying the least square method, a correction amount for the approximate value is obtained.
(Iv) The approximate value is corrected.
(V) The operations from (ii) to (v) are repeated using the corrected approximate value until convergence.

  Next, details of the orientation work by the orientation unit 24 will be described. There are three modes of orientation, manual, semi-automatic orientation mode, and automatic orientation mode, and they are selected as appropriate according to the arrangement state of corresponding points in the orientation image. FIG. 11 is a flowchart of the orientation work in the manual and semi-automatic orientation modes. First, the orientation unit 24 enters the orientation mode (S100). At this time, the operator designates an image to be standardized. In this case, there is no particular limitation on the designated number of images. Then, the measurement object image designated by the orientation unit 24 is displayed on the screen of the display device 40 (S110).

  The operator sets the orientation setting to semi-automatic orientation mode or manual (S120). In this case, a manual may be selected as the default of the orientation work by the orientation unit 24. The semi-automatic orientation mode has both a gravity center detection mode and a corner detection mode, and details will be described later. Next, for example, as shown in FIGS. 5 and 6, the operator designates a point to be used for image orientation (S130). In this case, corresponding points on the left and right images of the measurement object 1 are instructed by the corresponding point instruction unit 22. In the case of a manual, the corresponding points on the left and right images of the measurement object 1 are designated on the image of the display device 40 with the mouse cursor. In the case of the semi-automatic orientation mode, the corresponding point (centroid or corner) is specified by the corresponding point instruction unit 22 by designating the vicinity of the corresponding point. Then, it is determined whether or not the corresponding point designation on the left and right images of the measurement object 1 has been completed (S140). If No, the corresponding point designation is continued by returning to S130, and if Yes, the corresponding point is returned (S150). .

  Hereinafter, the algorithm of the semi-automatic orientation mode will be described. FIG. 12 is an explanatory diagram of a retro target attached to the feature point of the measurement object. The retro target 200 is a mark (target) whose center position 202 can be clearly identified, and has an inner circle portion 204 and an outer circle portion 206 formed concentrically. Visibility is enhanced by increasing the brightness of the inner circle portion 204 and decreasing the brightness of the outer circle portion 206. Therefore, the retro target 200 is affixed to feature points that are particularly important for positioning, such as the center of gravity and corners of the measurement object 1. In addition, the brightness of the inner circle part 204 and the outer circle part 206 may be reversed in contrast between light and dark.

  FIG. 13 is a flowchart for explaining an algorithm for detecting the center of gravity in the semi-automatic orientation mode. First, the retro target 200 is pasted to the feature point of the measurement object 1 in advance. Alternatively, instead of the retro target 200, a feature point with high luminance on the measurement target 1 may be instructed by the corresponding point instruction unit 22. The operator designates the vicinity of the corresponding point with the mouse cursor on the image of the measurement object 1 (reference image) displayed on the image of the display device 40 (S200). Then, the corresponding point instruction | indication part 22 determines the presence range of a target from the vicinity of the corresponding point in a search image (S210).

  FIG. 14 is an explanatory diagram of the center-of-gravity position detection using a retro target, where (A1) is a retro target with a dark inner circle portion, (A2) is a light intensity distribution diagram in the diameter direction of the retro target of (A1), and (B1 ) Is a retro target with a bright lightness in the inner circle, and (B2) is a lightness distribution diagram in the diameter direction of the retro target of (B1). When the retro target has a bright inner circle as shown in FIG. 14 (A1), the reflected light amount at the center of gravity position in the captured image of the measuring object 1 is a bright portion, and the light amount distribution of the image is As shown in FIG. 14A2, it is possible to obtain the inner circle portion 204 and the center position 202 of the retro target from the threshold value T of the light amount distribution.

  Returning to FIG. 13, if the imaging state of the measurement object 1 is poor and the target range cannot be calculated in S210, an error display is displayed because the center of gravity position cannot be detected (S215). Then, the process jumps from S215 to S250 to determine whether or not the designated point near the corresponding point is to be corrected (S250). If Yes, the process returns to S200 and designates another point as the designated point near the corresponding point as a measurement point. In the case of No in S250, the mode is switched to the manual measurement mode, the designation is switched to another feature point, and the feature point position is measured again (S260).

Then, the center of gravity position is calculated by, for example, the moment method by the orientation unit 24 or the corresponding point instruction unit 22 (S220). For example, the plane coordinates of the retro target 200 shown in (A1) of FIG. 14 are (x, y). Then, Expressions (6) and (7) are calculated for points in the x and y directions where the brightness of the retro target 200 is greater than or equal to the threshold value T.
xg = {Σx * f (x, y)} / Σf (x, y) (6)
yg = {Σy * f (x, y)} / Σf (x, y) (7)
Here, (xg, yg) is a coordinate of the center of gravity position, and f (x, y) is a lightness value on the (x, y) coordinate. In the case of the retro target 200 shown in (B1) of FIG. 14, the above equations (6) and (7) are calculated for points in the x and y directions whose brightness is equal to or less than the threshold value T.

  The orientation unit 24 or the corresponding point instruction unit 22 displays the corresponding point position obtained in S220 on the screen (S230). If the corresponding point position displayed in S230 conforms to the case where the corresponding point position is applied to the operator or a predetermined criterion, the orientation work is terminated and returned (S240). On the other hand, if the corresponding point position displayed in S230 is incompatible when applied to the operator or a predetermined criterion, or if the center of gravity position cannot be detected and an error is displayed, the process jumps to S250 described above. .

  FIG. 15 is a flowchart for explaining a corner detection algorithm in the semi-automatic orientation mode. The corner detection algorithm is applied when there is a corner where straight lines intersect on the image of the measurement object 1 or when a target having an orthogonal straight line is attached to the measurement object 1. For example, when the measuring object 1 is a building having many straight lines and corners such as a building, it is extremely effective.

  The operator designates the vicinity of the corresponding point with the mouse cursor of the corresponding point instruction unit 22 on the image (reference image) of the measurement object 1 displayed on the image of the display device 40 (S300). Then, the corresponding point instruction unit 22 automatically sets a designated point of the reference image and a search area (area) in the vicinity of the corresponding point in the search image (S310). In this case, since the feature point detection is designated as corner detection, the search area is set as a constant in advance by default according to the image scale of the measurement object 1. Then, edge detection in the search area is performed by the orientation unit 24 or the corresponding point instruction unit 22 (S320). As for edge detection, for example, an edge detection filter or a LOG filter (described below) can be used. However, any detection method used for image edge detection in image processing may be used.

An example of edge detection will be briefly described below. An image of L × L pixels centering on the detection point is set as a target image. Apply Laplacian-Gaussian filter (LOG filter), which is the second derivative of the Gaussian function shown in equation (8), to the target image grayscale waveform, and detect two zero crossing points, that is, edges of the curve in the calculation result by subpixels To do.
∇ ² · G (x) = (x ² −2σ ² / 2πσ ⁶ ) · exp (−x ² / 2σ ² ) (8)
Here, σ is a parameter of the Gaussian function.

  Subsequently, the corresponding point instruction unit 22 performs straight line detection on the detected edge (S330). The corresponding point designating unit 22 or the operator sees the continuity of the edges and fits two edges having the most continuity to a straight line. The straight line detection method is not limited to this, and any method may be used. And the corresponding point instruction | indication part 22 calculates | requires the intersection about two detected straight lines (S340). Then, the corresponding point instruction unit 22 displays the result on the image of the display device 40 (S350). It is determined whether or not the result of the feature point detected at the corner is suitable when the result is applied to an operator or a predetermined criterion (S360). If Yes in S360, it is determined whether there is a next corresponding point (S380). If there is a next corresponding point in S380, the next corresponding point is designated and the process returns to S300. If there is no next corresponding point in S380, the corner detection process is terminated and returned (S390).

  If No in S360, it is determined whether or not the designated point near the corresponding point is to be corrected with the mouse cursor (S370). If Yes, the process returns to S300 and designates another point as the designated point near the corresponding point. . If No in S370, the measurement is switched to the manual measurement mode and remeasured (S375).

  Next, the case of automatic measurement will be described. FIG. 16 is a flowchart for explaining the algorithm of the automatic orientation mode. In the automatic orientation mode, the retro target 200 (see FIG. 12) is pasted on the feature point of the measurement object 1 in advance. Next, the operator sets an automatic orientation mode in the orientation unit 24 (S400). Then, the measurement object image is displayed on the image of the display device 40 (S410). Next, the corresponding point instruction unit 22 registers one corresponding point image as a template (S420). Subsequently, the corresponding point instruction unit 22 searches for the same retro target image as the image registered in the template image on the measurement target image (S430). In this case, an image search is performed by template matching.

  Here, the details of template matching will be described. Template matching includes a normalized correlation method, a residual sequential test method (SSDA method), and other various calculation principles. If a residual sequential test method is used as template matching, the processing can be speeded up. Here, the residual sequential test method will be described.

処理の高速化をはかるため、式（９）の加算において、R(a,b)の値が過去の残差の最小値を越えたら加算を打ち切り、次の画素座標(a,b)に移るよう計算処理を行うと良い。 FIG. 17 is an explanatory diagram of an input image and a template image in the residual sequential test method. By performing the residual sequential test method provided in the corresponding point designating unit 22, a template image of N1 × N1 pixels is converted to a larger search range (M1−N1 + 1) ² in the input image of M1 × M1 pixels. move. The position of the template image is represented, for example, by coordinates (a, b) in the input image of the representative point provided at the upper left corner. Then, the corresponding point instruction unit 22 calculates the residual R (a, b) of Equation (9) at each movement position of the template image, and obtains the minimum movement position. The moving position where the residual R (a, b) is minimum is the position of the image obtained by template matching.

In order to speed up the processing, in the addition of Expression (9), if the value of R (a, b) exceeds the minimum value of the past residual, the addition is aborted, and the process proceeds to the next pixel coordinate (a, b). It is better to perform a calculation process.

  Returning to FIG. 16, the center-of-gravity positions are detected in more detail for all the targets searched in S430 (S440). For detecting the position of the center of gravity, for example, the moment method described above may be used. The orientation unit 24 or the corresponding point instruction unit 22 displays the target position obtained in S440 on the screen (S450). If the corresponding point position displayed in S450 conforms to the case where it is applied to the operator or a predetermined criterion, the orientation work is terminated and returned (S460). On the other hand, if the target position displayed in S450 is incompatible when applied to the operator or a predetermined criterion, or if the center of gravity position cannot be detected and an error is displayed, the semi-automatic orientation mode or manual The mode shifts to the non-conforming target position (S470).

[Create model image]
Next, model images used in the model will be described. The model image is used in the model forming unit 32, the image corresponding unit 34, the model display unit 35, and the like. FIG. 18 is a flowchart for explaining a model image creation procedure. First, coordinate transformation parameters are obtained using coordinates obtained by the orientation work and orientation calculation processing. That is, the image coordinates measured by the orientation work are associated with the ground coordinates calculated by the orientation calculation process (in the case of a model coordinate system, the ground coordinates are temporarily set) to obtain a coordinate conversion parameter (S500).

ここで、（ｘ、ｙ）は画像座標、（Ｘ、Ｙ、Ｚ）は地上座標、Ｌ１〜Ｌ１１は未知変量である。３次の射影変換式（１０）を、基準点のデータに基づき最小二乗法を用いて解くと、画像座標（ｘ、ｙ）と３次元座標（Ｘ、Ｙ、Ｚ）との関係を決定する各種変換パラメータを取得することができる。 The coordinate conversion parameter is obtained by the following cubic projective transformation formula (10).

Here, (x, y) are image coordinates, (X, Y, Z) are ground coordinates, and L1 to L11 are unknown variables. When the cubic projective transformation equation (10) is solved using the least square method based on the reference point data, the relationship between the image coordinates (x, y) and the three-dimensional coordinates (X, Y, Z) is determined. Various conversion parameters can be acquired.

ここで、（Ｘ_０、Ｙ_０）は地上座標系でのモデル画像の左上の位置、（ΔＸ、ΔＹ）は地上座標系での１画素の大きさ（例：ｍ／ｐｉｘｅｌ）、（ｘ、ｙ）はモデル画像の画像座標、（Ｘ、Ｙ、Ｚ）は地上画像、係数ａ、ｂ、ｃ、ｄはある画像座標（ｘ、ｙ）を内挿する複数の基準点により形成される平面方程式の係数である。 Next, the ground coordinates of each pixel (pixel) on the model image are calculated (S510). In this process, the image coordinates (x, y) of the model image are converted to ground coordinates (X, Y, Z) in order to create a model image. The ground coordinates (X, Y, Z) are calculated using the conversion parameters previously obtained in S500 of the coordinate conversion parameter calculation process. That is, the ground coordinates (X, Y, Z) corresponding to the image coordinates (x, y) of the model image are given by the following equation (11). In this way, the acquisition position of each pixel on the model image can be obtained.

Here, (X ₀ , Y ₀ ) is the upper left position of the model image in the ground coordinate system, and (ΔX, ΔY) is the size of one pixel in the ground coordinate system (eg, m / pixel), (x, y) is an image coordinate of the model image, (X, Y, Z) is a ground image, and coefficients a, b, c, and d are planes formed by a plurality of reference points that interpolate a certain image coordinate (x, y). Is the coefficient of the equation.

  This time, using the transformation parameters obtained in step S500, the image coordinates (x, y) corresponding to the ground coordinates (X, Y, Z) obtained in step S520 by the cubic projective transformation equation (10). ) Is calculated (S520). From the image coordinates (x, y) thus determined, the density value on the ground coordinates (X, Y, Z) of the corresponding image is acquired. This density value is the density of the pixel at the two-dimensional position (X, Y) on the model image. In this manner, the image density to be pasted at the position (X, Y) on the ground coordinates is acquired. Image pasting is performed by performing the above processing on all the pixels of the model image (S530).

[Model image formation with different viewpoints]
Next, the calculation principle relating to the rotation of the model image performed by the coordinate conversion unit 37 will be described. As described in S <b> 92 of FIG. 2, when the orientation of the model of the measuring object 1 is instructed by the orientation instructing unit 36, the coordinates are converted by the coordinate converting unit 37 and the orientation indicated by the model display unit 35 is determined. Since the image with the three-dimensional texture of the measurement object 1 is displayed, the image with the three-dimensional texture in any direction of the measurement object 1 is obtained. Therefore, the principle of creating a model image viewed from a different viewpoint will be described. This principle is to form a model image after rotating the coordinate system by the coordinate conversion unit 37 in the direction instructed by the posture instruction unit.

  FIG. 19 is an explanatory diagram of the principle of model image formation with different viewpoints. As shown in FIG. 19, by rotating X, Y, and Z of the ground coordinate system to ω, φ, and κ in the respective axial directions, ground coordinate systems X ′, Y ′, and Z ′ with different viewpoints are obtained. It is done. Therefore, an image is pasted on the ground coordinate systems X ′, Y ′, and Z ′ whose viewpoint has been changed to create a model image.

ここで回転行列Ｒには次の式（１３）、（１４）が成立している。

ここで、（Ｘｏ，Ｙｏ，Ｚｏ）は投影中心の座標である。 This will be described in detail below. For example, an object P (X, Y, Z) represented by ground coordinates X, Y, Z without inclination is a ground coordinate system X ′, Y ′, Z ′ having inclinations (coordinates with different viewpoints). If the coordinates taken by the system) are P ′ (X ′, Y ′, Z ′), the coordinates can be obtained by the following equations (12) to (15).

Here, the following equations (13) and (14) are established in the rotation matrix R.

Here, (Xo, Yo, Zo) are the coordinates of the projection center.

ここで座標変換して求められた各座標Ｘ’、Ｙ’、Ｚ’について、モデル画像形成処理を行なえば、視点を変えたモデル画像を得ることができる。 Further, an element a _ij (i = 1 to 3, j = 1 to 3) of the rotation matrix R is expressed by the following equation (15).

If a model image forming process is performed for each of the coordinates X ′, Y ′, and Z ′ obtained by the coordinate conversion, a model image with a different viewpoint can be obtained.

  According to the three-dimensional image display apparatus of the present invention, the stereo image of the measurement object and the model formed by the model forming unit are associated by the image corresponding unit using the corresponding point relationship obtained by the orientation unit, and the model Since the display unit is configured to display an image with a three-dimensional texture of a measurement object using a stereo image associated with a model, a stereoscopic device is also used for stereo measured 3D data. Thus, the 3D data and the image of the measurement object 1 can be easily confirmed without performing stereoscopic viewing.

  Further, as in the embodiment, a computer having an information processing processor such as a PC (personal computer) as a three-dimensional image display device, a display device such as an LCD (Liquid Crystal Display) monitor, and a three-dimensional image installed on the PC If there is software for a display device and a photographic device to be calibrated such as a digital camera that shoots a stereo image of a measurement object, 3D measurement data acquired from the stereo image is integrated with an image with a three-dimensional texture of the measurement object. Can be visualized. Therefore, a system can be constructed at low cost by using a general-purpose and low-cost computer and monitor device without using an expensive and precise stereoscopic device that has been conventionally required for 3D data measured in stereo.

1 is an overall configuration block diagram illustrating a first embodiment of the present invention. 3 is a flowchart for explaining the operation of the three-dimensional image display apparatus shown in FIG. It is a screen figure explaining an example of the orientation result of an orientation calculation process. It is drawing explaining an example of the stereo measurement performed with respect to a stereo image. It is drawing explaining the other example of the stereo measurement performed with respect to a stereo image. It is an enlarged view of an image area 162L near the feature point and an image area 162R near the corresponding point. It is a figure which shows an example of the model of the measuring object 1, and has shown the wire frame surface. It is the figure which expressed the texture which expresses a three-dimensional feeling with the wire frame. In the texture mapping display example, the measurement object 1 is displayed as an image with a three-dimensional texture. It is explanatory drawing of the model coordinate system XYZ and camera coordinate system xyz in a stereo image. It is a flowchart of the orientation work of a manual and semiautomatic orientation mode. It is explanatory drawing of the retro target stuck on the feature point of a measurement object. It is a flowchart explaining the algorithm of the gravity center detection in semiautomatic orientation mode. It is explanatory drawing of the gravity center position detection using a retro target. It is a flowchart explaining the algorithm of the corner detection in semiautomatic orientation mode. It is a flowchart explaining the algorithm of automatic orientation mode. It is explanatory drawing of the input image and template image in a residual sequential test method. It is a flowchart explaining the preparation procedure of a model image. It is a principle explanatory view of model image formation which changed a viewpoint.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Measurement object 10 Measurement object image data storage part 12 Stereo image data storage part 16 Measurement object position data storage part 22 Corresponding point instruction | indication part 24 Orientation part 30 Display image formation part 31 Three-dimensional coordinate data part 32 Model formation part 34 Image correspondence unit 35 Model display unit 40 Display device

Claims (5)

A stereo image data storage unit for storing a stereo image of the measurement object;
An orientation unit that obtains a corresponding point relationship of the stereo image based on the photographing position and the inclination with respect to the stereo image;
A three-dimensional coordinate data unit for obtaining three-dimensional coordinate data of corresponding points of the measurement object from the corresponding point relationship obtained by the orientation unit;
A model forming unit that forms a model of the measurement object from the three-dimensional coordinate data of the corresponding points;
An image correspondence unit that associates the stereo image of the measurement object stored in the stereo image data storage unit with the model formed by the model formation unit using the corresponding point relationship obtained by the orientation unit;
A model display unit for displaying an image with a three-dimensional texture of the measurement object using a stereo image associated with the model by the image corresponding unit;
3D image display device. A posture instructing unit for instructing a posture of the model of the measurement object;
A coordinate conversion unit that performs coordinate conversion of the corresponding points in response to an orientation instruction to the model;
The three-dimensional image display device according to claim 1, wherein the model display unit displays an image with a three-dimensional texture of the measurement object according to a posture instructed by the posture instruction unit.   2. The image correspondence unit covers the stereo image by a unit image plane formed by a plurality of the corresponding points, and associates the model and the stereo image using the unit image plane. Or the three-dimensional image display apparatus of Claim 2. And a corresponding point indicating unit for indicating a corresponding point used in the orientation unit;
The corresponding point indicating unit is configured to indicate a feature point in the vicinity of the specified position with respect to the measurement object displayed on the display device that displays the stereo image. The three-dimensional image display device according to item. Obtaining a corresponding point relationship of the stereo image based on the shooting position and inclination of the stereo image of the measurement object by the orientation unit;
Obtaining three-dimensional coordinate data of corresponding points of the measurement object from a corresponding point relationship obtained by the orientation unit by means of a three-dimensional coordinate data unit;
Forming a model of the measurement object from the three-dimensional coordinate data of the corresponding point by a model forming unit;
A step of associating the stereo image of the measurement object stored in the stereo image data storage unit with the model formed by the model forming unit using the corresponding point relationship obtained by the orientation unit by the image corresponding unit. When;
Displaying a three-dimensional textured image of the measurement object using a stereo image associated with the model by a model display unit;
3D image display method for causing a computer to execute.

Images