JP2002230586A - Image system for supporting three-dimensional survey using composite reality - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image system for supporting three-dimensional survey for displaying a precise three-dimensional position in survey work by composite reality on a notebook-sized or portable personal computer (herein after referred as a PC) and a stereoscopic display. SOLUTION: A CCD camera in place of an eye of a worker W, a sensor for obtaining a viewpoint position, and a sensor for obtaining a viewpoint attitude are mounted on the PC, and a plurality of reference points which a three-dimensional coordinate is known are arranged in a space which is an observation object. An image of the CCD camera is accumulated in an image accumulating part, is used in compositing with a plot object by an image composite part, and is used for correction of an error of a viewpoint position and viewpoint attitude sensor by a position and attitude detecting means. Because the plot object is superimposed on a predefined coordinate in a real space which is the observation object on the basis of obtained position and attitude information of the CCD camera, a stereoscopic image is projected successively on the PC and the stereoscopic display by a plot object restructuring part for restructuring the plot object and an image signal converting part for converting image data of the restructured plot object into an image display signal which can be displayed on an image display device.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional surveying support image system using mixed reality, and more particularly, to a construction site where a virtual three-dimensional image projected on a real space and projected is used in construction work. The present invention relates to a three-dimensional surveying support image system that uses mixed reality to provide location information that conforms to the requirements of (1) and streamlines construction.

[0002]

2. Description of the Related Art With the recent development of computer technology,
Mixed Reality (Mixed Re) for the purpose of fusing electronic virtual data with a computer to the real space
(hereinafter referred to as MR). MR has been actively studied in various fields as a technique for constructing a virtual space in a computer and enhancing virtual reality (VR) for giving an experience as if it were present in the virtual space.

A method of experiencing this MR is to create a virtual space by CG (computer graphics) displayed on a display element incorporated on a half mirror positioned so as to transmit light from the outside entering through goggles. A transflective HMD (HeD) that can project a stereoscopic image through an optical system so as to be superimposed on incident light.
The method is classified into an optical see-through method using ad mounted display, and a video see-through method in which a stereoscopic image in a virtual space is superimposed on an image captured by a video camera.

One of the technical problems of MR is that there is a problem of misalignment between a real space and a virtual space caused by an error in detection or calculation of an operator's viewpoint position and viewpoint posture. Efforts have been made. The problem of alignment in MR is reduced to the problem of accurately determining the three-dimensional position and posture of the worker.

[0005] A method for obtaining a three-dimensional position and posture of an operator in MR includes an automatic tracking type total station mainly used in the case of an optical see-through type MR.
Tracker-based method using GPS receiver, gyro sensor, acceleration sensor, magnetic sensor, ultrasonic sensor, etc., and positioning on image based on image information, mainly used for video see-through type MR Is generally used.

[0006] In the tracker-based method, although the operation is stable, there are problems such as that the action range is limited to the effective range of the tracker and the measurement accuracy is not sufficient. It is the cause of misalignment.

On the other hand, the image-based method lacks system stability due to instability in image processing such as landmark extraction and identification. On the other hand, in recent years, there has been proposed a method of performing alignment using both a sensor and image information.

In construction works such as civil engineering and construction, the construction position of a structure is grasped in absolute coordinates, and a structure having a designed structural dimension is constructed based on the absolute coordinates. For this reason, a surveying operation for measuring three-dimensional absolute coordinates is required at a construction site. The positioning of such a structure has been carried out using a measuring device such as a distance measuring device, an angle measuring device, a level gauge, a steel tape or a laser marker, and a great deal of labor and work time has been required for the actual measurement.

[0009]

Many of the research cases relating to the registration based on the image-based method in MR are methods for two-dimensionally matching the displacement with the appearance of the worker. The correction is not dimensionally correct and is limited to the case where the sensor error is small.

Therefore, an object of the present invention is to solve the above-described problems relating to the positioning in MR, and to allow a worker to view virtual stereoscopic video information created by CG via a display, thereby enabling the operator to work in real space. The purpose of the present invention is to provide useful information to a surveying operation performed by a worker to improve the efficiency of the operation, the construction accuracy and the safety.

[0011]

SUMMARY OF THE INVENTION A three-dimensional surveying support image system using mixed reality according to the present invention is displayed on a notebook or portable personal computer via a CCD camera as an eye for an operator. The image of the observation target survey area obtained is a three-dimensional survey in which, in addition to the real space, a drawing object, which is a pre-designed structure, is superimposed on the virtual space at the predetermined installation location to provide an image display that provides mixed reality. An image system for assistance, wherein the personal computer mounts left and right eye CCD cameras at predetermined positions, sequentially obtains camera viewpoint position and posture sensor data, and transmits the sensor data to a sensor data storage unit. The CCD camera, which includes a plurality of reference points whose three-dimensional coordinates indicating a predetermined installation position of the drawing object are mounted, wherein the sensor acquisition device is mounted at a predetermined position. Image storage means for sequentially writing and updating the image as a digital image in the image storage unit in the image storage unit, and completion of the initial setting in which the operator extracts the photograph coordinates of the reference point on the initial image and corrects at least the principal point position and the screen distance of the camera. Subsequently, a plurality of reference points in each image of the left and right cameras called from the image storage unit are automatically extracted on the image plane, and the reference point automatic extraction means for determining the position and the three-dimensional coordinates are (X, Y, Z). ), The photo coordinates (x,
In a collinear conditional expression based on projective geometry representing y),
The reference point coordinates for which the coordinates (X, Y, Z) are known, and the respective photograph coordinates (x,
y), the unknown CCD camera position (coordinates X ₀ ,
Y ₀ , Z ₀ ) and posture (rotational axes ω, φ, κ) based on position / posture detection means, and the position and posture of the CCD camera mounted on the personal computer obtained by the position / posture detection means. A drawing object reconstructing means for sequentially reconstructing a drawing object in order to call and superimpose the drawing object from its database in accordance with the photograph coordinate reference point in the real space; and a CCD camera image to be observed is called from the image storage unit. Image synthesizing means for superimposing the virtual space of the reconstructed drawing object on the real space, and an image for converting image data obtained by synthesizing the drawing object on the CCD camera image into an image display signal which can be displayed on the display unit of the personal computer And at least signal conversion means.

Further, when solving the camera position and orientation as unknowns in the collinear condition expression of the position / posture detection means, to simplify the correction calculation including the displacement of the principal point,
After correcting the internal orientation elements of at least principal point displacement and screen distance by stereo pair images of two CCD cameras, the unknown is decomposed into its approximate value and a small correction amount, and a nonlinear equation is linearized and solved by iterative calculation. It is characterized by the following.

[0013] The position / posture detecting means may be provided in an initial state after the initial setting by the operator.
Acquire the sensing result of the position and orientation sensor acquisition device section of the personal computer, solve the collinear conditional expression as the approximate value of the camera position and orientation that gives the value in the initial state, and the approximate values of the subsequent frames are It is characterized in that the sensing result at the time is used, and if the error is large, the analysis result of the previous frame is used.

Further, the position data acquisition device section mounts a reflecting prism at a predetermined position on a personal computer, and transfers position data detected by a position measuring total station having an automatic tracking function to the reflecting prism via a wireless modem. The data is transmitted and sequentially recorded in the sensor data storage unit.

Further, the attitude data acquisition device section is characterized in that a gyro sensor is mounted at a predetermined position of a personal computer, and the attitude data detected there is sequentially recorded in the sensor data storage section.

The personal computer has a liquid crystal display as a display unit, and the left and right images of the left-eye CCD camera and the right-eye CCD camera are refreshed and displayed at a constant frequency on the left and right sides thereof. It is characterized by experiencing mixed reality by wearing liquid crystal shutter glasses that open and close.

Also, the image display signal converted by the image conversion means is transmitted to the display section of the personal computer and also to the display section of the stereoscopic display, and two image splitters are arranged to separate the left and right images. The display is characterized in that the worker can view stereoscopically without glasses.

Further, the reference point automatic extracting means includes a plurality of the plurality of reference points based on camera position and orientation data in the previous frame, in addition to the CCD camera principal point position and screen distance corrected in the initial image. A search range in which the photo coordinates on the camera image are calculated backward, a predetermined search range is set around the coordinates, and the accurate photo coordinate of the reference point is automatically extracted based on the RGB information of the reference point there. It is characterized by including setting means.

Further, the reference point automatic extracting means includes a reference point marking means for marking in advance a color different from the surroundings of the reference point, and an operator sets the first frame of the initial image in the initial image before executing the reference point automatic extraction. It is characterized in that the photograph coordinates of the reference point can be easily extracted.

In the automatic pattern recognition for the RGB values within the search range at the time of the reference point extraction by the reference point automatic extracting means, the following four processes are repeated. R value of reference point i extracted in the previous frame (R
_i (t)), G value (G _i (t)), B value (B
_i (t)), width _(w i _(t)), stores the height _(h i _(t)); "truncation" _{(1) R i (t)} , G
A value (A means any one of R, G, and B) that takes the maximum value (A _i (t) _MAX ) of _i (t) and B _i (t)
There _{A i (t + 1) <} A i and the pixel is _{(t) MAX / 2, (} 2) R (GB) value respectively threshold dR (G
B), and R (GB) _i (t) −dR (GB) <R
(GB) _i (t + 1) <R (GB) _i (t) + dR (G
B) Pixels equal to or larger than B) are discarded (weight is set to 0); When there are two or more sets of pixels having a range of weight ≠ 0 in the “continuity processing” search range, the width w of the set is
_i (t + 1) and the height _hi (t + 1) are w
_i (t)) and only those close to _hi (t) are extracted; weighted barycentric coordinates for the A value within the search range are obtained and set as photograph coordinates.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a three-dimensional surveying support image system utilizing mixed reality according to the present invention will be described below with reference to the drawings.

FIG. 1 shows a notebook or portable personal computer 1 used in an image system 100 for supporting three-dimensional surveying according to the present invention.
FIG. 2 is a diagram for explaining the flow of operation of the system 100.

1 and 2, reference numeral 1 denotes a personal computer, 2 denotes a stereoscopic display, and P _{1 to} P _{n denote} observation surveying areas 1.
There are a plurality of reference points indicating the installation locations of components to be installed at 0. The three-dimensional coordinates (X, Y, Z) of this reference point are known.

The personal computer 1 is controlled by a commercially available operating system (OS), where 3 is a central control unit, and 4 is a display unit or a liquid crystal display. 5 is a key and pointing device input unit, 6 is a CD disk drive, 7 is a hard disk drive, 8 is a bus,
/ F indicates an interface circuit.

Reference numerals 11L and 11R denote CCD cameras for the left and right eyes, respectively, which are mounted at predetermined positions on the housing of the personal computer 1 as shown in FIG.
Including the reference point P ₁ to P _n of the set location of the planned installation structure to get the video.

Reference numerals 21 and 22 denote position and orientation sensor acquisition units, which are mounted at predetermined positions on the housing of the personal computer 1 as shown in FIG. 2, and acquire the viewpoint position and orientation sensing results of the camera.

Reference numeral 42 denotes a video signal transfer unit. The video signal generated by the video signal conversion means 41 described later is sent to the video signal receiving unit 40 of the stereoscopic image display 2 via the transfer unit 42 and is sent to the stereoscopic image display unit 2. The information is displayed on the display unit 44 of the display 2.

Next, the system 1 of the present invention will be described with reference to FIG.
00 will be described in detail.

CCD camera image storage means 12 necessary for operation of the system, camera image reference point automatic extraction means 1
3. The application programs of the camera position / posture detecting means 23, the drawing object reconstructing means 31, the image synthesizing means 32, and the video signal converting means 41 are recorded in the hard disk device 7 or the like in advance.

Each of the above means 12, 13, 23, 31, 3
2, 41 will be described in the order of the operation flow of the system 100.

First, the CCD cameras 11L and 11R attached to the personal computer 1 are set facing the observation target survey area 10. The reference point P ₁ to P _n indicating the location for installing the pre-designed structure installing the computer 1 to enter into the real space image of the CCD camera.

Next, the image is acquired from the CCD camera by the image storage means 12. The CCD camera image includes a plurality of reference points whose three-dimensional coordinates indicating the setting and installation positions of the drawing object which is a pre-designed structure are known, and the camera image is sequentially converted into a digital image in real time. The data is updated in the image storage unit 12a.

Next, the left and right CCs are displayed on the display unit 4 of the personal computer 1.
The images acquired from the D cameras 11L and 11R are displayed on the left and right, respectively. The operator determines the reference point P on this initial image.
_{After the} “initial setting” of extracting the photograph coordinates (x, y) of _{1 to} P _n and correcting the principal point position of the camera 11 and the screen distance is completed, the reference point automatic extraction means 13 causes the image storage unit 12 a to be used. automatically extracting a plurality of reference points P ₁ to P _n in each image of the left and right cameras more calls in the image plane, determining its position.

Next, the position / posture detecting means 23
In a collinear conditional expression based on projective geometry representing photographic coordinates (x, y) of a point whose dimensional coordinates are (X, Y, Z), a reference point coordinate whose coordinates are known, From the photograph coordinates (x, y) of the plurality of reference points automatically extracted and calculated, the unknown CCD camera position (coordinate X
₀ , Y ₀ , Z ₀ ) and attitude (rotational axes ω, φ, κ).

Here, the aforementioned collinear condition expression will be described in more detail.

When the distortion of the camera lens is negligible as in a measuring camera, the photographic coordinates (x, y) of a point whose coordinates are (X, Y, Z) are expressed by the collinear conditional expression ( 1) and (2). (See Fig. 3)

[0037]

(Equation 1)

[0038]

(Equation 2)

Here, c is the screen distance, (X ₀ , Y ₀ , Z
₀ ) is the coordinates of the camera projection center, (ω, φ, κ) is the attitude of the camera, and a _ij is an element of the rotation matrix. When the screen distance c of the camera is known, the camera position and orientation can be uniquely obtained if three or more reference points are given. However, the CCD camera used is a non-measurement camera,
It is necessary to consider the displacement (x _h , y _h ) of the principal point position H and the distortion of the camera lens. In this case, the equation (1)
Left side of the collinear conditional expression (2), respectively _{x-x h} - △
x, y-y _h- △ y. However,

[0040]

(Equation 3)

[0041]

(Equation 4)

[0042]

(Equation 5)

Is as follows. Further, since the screen distance c of the camera is also strictly unknown, the collinear conditional expression includes the internal orientation element (c) in addition to the external orientation elements (X ₀ , Y ₀ , Z ₀ , ω, φ, κ). , X _h , y _h , k ₁ , k ₂ , k ₃ , γ, p ₁ ,
p ₂ ) also becomes an unknown number, and it takes time for iterative calculation. Therefore, in order to easily calculate the principal point position H correction,
2 previously worn on the subject's HMD under the same conditions as the MR experience
These internal orientation elements are calibrated by stereo pair images of the two CCD cameras 11L and 11R. Since the collinear conditional expressions (1) and (2) are nonlinear with respect to the unknowns (camera position and attitude), the analysis decomposes the unknowns into “approximate values” and “small correction amounts” to linearize the nonlinear equations, Find a solution by iterative calculation.

In the position / posture detecting means 13, in the initial state after the above-mentioned “initial setting” is performed by the operator, the position / posture sensor acquisition device units 21, 22 of the personal computer, that is, the sensing results of the tracker are read. The collinear condition formulas (1) and (2) are obtained as the "approximate values" of the camera position and orientation which are acquired and given in the initial state.
That is, external orientation elements (elements related to camera position / posture, X ₀ , Y ₀ , Z ₀ , ω, φ, κ) given in the initial state
The "approximate value" uses the tracker sensing result, and the subsequent "approximate value" takes into account the accuracy of the tracker that can be used at that time. The tracker sensing result at that time is used. At this time, each external orientation element is individually given a weight to perform an analysis in consideration of the accuracy of the tracker. For example, if the deviation of the viewing angle called drift around the vertical axis of the gyro sensor is remarkable, κ
If the position sensor goes out of the measurement range, X ₀ ,
The weights of Y ₀ and Z ₀ will be reduced.

FIG. 3 is an explanatory view showing the collinear condition of the overlapped photograph, where P (X _P , Y _P , Z _P ) is a reference point, O ₁ , O ₁ ,
O ₂ is (X, Y, Z) is the left and right camera position, O
_C1 and O _C2 are left and right camera images P _C1 and P _C , respectively.
_C2 indicates photographic coordinates. H ₁ and H ₂ indicate respective principal point positions, and C ₁ and C ₂ indicate respective screen distances. Further, ω, φ, and κ determine the rotation angles of the coordinates X, Y, and Z axes, that is, the camera attitude.

Next, the drawing object reconstructing means 31
Calls the drawing object basic shape data unit 30 (database) in advance in accordance with the photographic coordinate reference point in the real space based on the position and orientation of the CCD camera mounted on the personal computer obtained by the position / posture detection means 23 described above. Then, the drawing object is sequentially reconstructed in order to superimpose the drawing object.

Next, the image synthesizing means 32 calls the CCD camera image to be observed from the image storage section 12a, and superimposes the virtual space of the reconstructed drawing object on the real space.

Next, the video signal conversion means 41 converts the image data obtained by combining the drawing object with the CCD camera image into a video display signal that can be displayed on the display unit 4 of the personal computer 1.

The converted output is output to the display unit 4 of the personal computer 1 and displayed. As described above, the data is received by the receiving unit 40 of the stereoscopic image display 2 via the transfer unit 42 and is displayed on the display unit 44.

Next, the above-mentioned reference point automatic extracting means 13 will be described in more detail.

The automatic extraction means 13 determines in advance the reference points P ₁ -P
Reference point marking means 13y for marking the periphery of _n with a different color so that the operator can easily extract the photograph coordinates of the reference point of the first frame in the initial image before executing the automatic extraction of the reference point. .

Further, the automatic extraction means 13 uses the initial image
Corrected CCD camera principal point position H ₁, H₂, Screen distance
C₁, C₂In addition to the camera position and figure in the previous frame
Reference point P based on current data₁~ P_nPhoto on camera image
Reverse the coordinates and set a predetermined search range around the coordinates
Then, based on the RGB information of the reference point there,
Search range setting means 13x for automatically extracting the photograph coordinates
Including. Only the parts that are required efficiently by this means 13x
By searching, it can be accurately extracted at high speed.

In the search range setting means 13x of the automatic extraction means 13, RGB in the search range at the time of extraction of the reference point is extracted.
In the automatic pattern recognition for the value, the R value (R
_i (t)), G value (G _i (t)), B value (B
_i (t)), width _(w i _(t)), stores the height _(h i _(t)). (T is the time of the previous frame) [Truncation processing] (1) R _i (t), G _i (t),
A value (A means any one of R, G, and B) that takes the maximum value (A _i (t) _max ) of B _i (t) is A
a pixel where _i (t + 1) < _Ai (t) _max / 2;
(2) A threshold value dR (GB) is provided for each R (GB) value, and R (GB) _i (t) −dR (GB) <R (GB)
Pixels other than pixels satisfying _i (t + 1) <R (GB) _i (t) + dR (GB) are rounded down (the weight of the rounded down pixel is set to 0). (Continuity processing) A set of pixels in which the range of weight ≠ 0 (the weight of the pixels that have not been truncated) continues in the search range is 2
If there are more than two, the width w _i (t + 1) and the height h of the set
_i (t + 1) is the most _w i _(t), to extract only those close to the _h i (t). The weighted barycentric coordinates for the A value within the search range are obtained and set as photograph coordinates. Is repeated.

Specific Embodiment of the Invention A three-dimensional surveying support image system 200 will be described below with reference to FIG.

Reference numeral 1 denotes a so-called notebook or portable personal computer 1 on which a CCD camera 11, a reflecting prism 24 as a position sensor, and a gyro sensor 25 as an azimuth sensor are attached.

Reference numeral 26 denotes an automatic tracking type total station, which sequentially transmits position data to the reflecting prism 24 via a wireless modem 27.

The position data detected by the total station 26 and the attitude data detected by the gyro sensor 25 are sequentially written and updated in the image storage unit 12a in real time.

On the other hand, the images of the CCD cameras 11L and 11R are sequentially stored as digital images in the image storage unit 12 in the personal computer 1, and on the personal computer 1, the operator first extracts the photographic coordinates of the reference point P on the initial image. a camera calibration i.e. X _h, subjected to principal point position and screen distance compensation of Y _h, sequentially after the completion of the initial settings, the automatic extraction and camera position and posture detection of the reference points is carried out automatically.

Based on the position and orientation of the camera detected by the above means, the drawing object is reconstructed by the drawing object reconstructing section 31 in the personal computer 1 and the image compositing section 32 is reconstructed.
The drawing object is superimposed on the CCD camera image, and is displayed on the display unit 4 of the personal computer 1 from the video signal conversion unit 41, and is received from the receiving unit 40 of the stereoscopic display 2 via the video signal transfer unit 42 of the personal computer 1. The data is transferred to the display unit 44 and displayed.

The display unit 4 is a liquid crystal display of the personal computer 1 and displays two different left and right images refreshed at a certain frequency, and the operator wears liquid crystal shutter glasses 43 which open and close in synchronization with this frequency. By doing so, you can experience mixed reality.

The display section 44 of the stereoscopic display 2 is a display on which two image splitters are arranged to separate the left and right images, so that an operator can perform stereoscopic viewing without glasses.

[0062]

As described above, according to the present invention, the following effects can be obtained.

That is, the accuracy of the positioning in the presentation of the mixed reality is improved, and in particular, by utilizing the virtual stereoscopic video projected on the real space in the construction work,
Providing location information suitable for the target site, it has the effect of speeding up surveying and construction, streamlining, and improving safety.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a three-dimensional surveying support image system according to the present invention.

FIG. 2 is an explanatory diagram of an operation flow of the three-dimensional survey support image system of the present invention.

FIG. 3 is an explanatory diagram of a collinear condition of a duplicate photograph.

FIG. 4 is a diagram showing a flow of an operation in a specific embodiment of the three-dimensional survey support image system of the present invention.

[Explanation of symbols]

Reference Signs List 1 notebook or portable personal computer 2 stereoscopic display 3 central control unit 4 personal computer display unit 5 key and pointing device input device 6 optical disk drive device 7 hard disk device 8 bus 10 observation surveying area 11, 11L, 11R CCD camera DESCRIPTION OF SYMBOLS 12 Image storage means 12a Image storage part 13 Reference point automatic extraction means 13x Extraction search range setting means 13y Reference point marking means 21 Position sensor acquisition device part 22 Attitude sensor acquisition device part 23 Position / posture detection means 23a Sensor data storage part 30 Drawing Object basic shape data part (database) 31 Drawing object reconstructing means 32 Image synthesizing means 40 Video signal receiving part of stereoscopic display 2 41 Video signal converting means 42 Video signal transfer part 43 Liquid crystal shutter glasses 44 Stand Visual imaging system for display units 100 and 200 three-dimensional surveying assistance display 2 I / F interface circuit portion _{P, P 1, P 2,} ‥‥, P n 3 -dimensional coordinates known reference point

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kazunobu Matsumoto 2 Sanbancho, Chiyoda-ku, Tokyo Tobishima Construction Co., Ltd. (72) Inventor Hirotaka Nakahara 2nd Sanbancho, Chiyoda-ku, Tokyo Tobishima (72) Inventor Masayuki Tsutsui 2nd Sanbancho, Chiyoda-ku, Tokyo Tobishima Construction Co., Ltd. (72) Inventor Yukiki Kumagai 2nd Sanbancho, Chiyoda-ku, Tokyo F-term in the formula company (reference) 5B050 BA04 BA08 BA09 BA10 BA11 EA07 EA19 EA27 FA02 FA06 5C061 AA03 AA20 AB04 AB12 AB14 AB20

Claims (10)

[Claims] 1. An image of an observation target survey area displayed on a notebook or portable personal computer via a CCD camera serving as an eye of an operator, in addition to its actual space, a structure designed in advance. 3. A three-dimensional surveying support image system for displaying a drawing object, which is an image display that superimposes a virtual space at a predetermined installation place on the virtual object and gives mixed reality, wherein the personal computer moves the left-eye and right-eye CCD cameras at predetermined positions. And a position and orientation sensor acquisition unit for sequentially acquiring and transmitting the camera viewpoint position and orientation sensor data to the sensor data storage unit are attached to a predetermined position, and three-dimensionally indicating a predetermined installation location of the drawing object. An image to be written and updated in real time as a digital image of the CCD camera including a plurality of reference points whose coordinates are known as a digital image. After the completion of the initial setting in which the operator extracts the photo coordinates of the reference point on the initial image and corrects at least the principal point position of the camera and the screen distance, the operator selects each of the left and right camera images called from the image storage unit. A reference point automatic extracting means for automatically extracting a plurality of reference points on an image plane and determining the positions thereof; and representing photograph coordinates (x, y) of a point whose three-dimensional coordinates are (X, Y, Z). In the collinear conditional expression based on the projective geometry, the reference point coordinates for which the coordinates (X, Y, Z) are known and the photograph coordinates (x, y) of the plurality of reference points automatically extracted and calculated are used. Position / posture detecting means for solving unknown CCD camera positions (coordinates X ₀ , Y ₀ , Z ₀ ) and postures (rotational axes ω, φ, κ), and mounting on the personal computer obtained by the position / posture detecting means Based on the CCD camera position and attitude Drawing object reconstruction means for sequentially reconstructing a drawing object in order to call and superimpose the drawing object from its database in accordance with the photograph coordinate reference point in the real space, and call a CCD camera image to be observed from the image storage unit, Image synthesizing means for superimposing the virtual space of the reconstructed drawing object on the real space; and converting image data obtained by synthesizing the drawing object with the CCD camera image into a video display signal that can be displayed on the display unit of the personal computer. An image system for supporting three-dimensional surveying using mixed reality, comprising at least video signal conversion means. 2. In solving the camera position and orientation as unknowns in the collinear conditional expression of the position / posture detection means,
In order to simplify the correction calculation including the displacement of the principal point, after correcting at least the internal orientation elements of the displacement of the principal point and the screen distance with the stereo pair images of the two CCD cameras, the unknown is approximated to its approximate value and minutely corrected. 2. The three-dimensional surveying support image system using mixed reality according to claim 1, wherein the nonlinear equation is linearized by being decomposed into quantities and solved by iterative calculation. 3. The position / posture detecting means, in an initial state after the operator performs the initial setting, acquires a position / posture sensor acquisition device unit sensing result of the personal computer and initializes the value. Solve the collinear conditional expression as the approximate value of the camera position and orientation given in the state, use the sensing result at that time as the approximate value of the subsequent frames, and use the analysis result of the previous frame if the error is large 3. The three-dimensional surveying support image system using mixed reality according to claim 2, wherein: 4. The position data acquisition device section mounts a reflection prism at a predetermined position on a personal computer, and transmits position data detected by a position measurement total station having an automatic tracking function to the reflection prism via a wireless modem. The image system for supporting three-dimensional surveying using mixed reality according to claim 1, 2 or 3, wherein the image data is transmitted and sequentially recorded in the sensor data storage unit. 5. The attitude data acquisition device section, wherein a gyro sensor is mounted at a predetermined position of a personal computer, and the attitude data detected there is sequentially recorded in the sensor data storage section. , 3 or 4, a three-dimensional surveying support image system using the mixed reality. 6. The personal computer has a liquid crystal display as a display unit, and displays left and right images of a left-eye CCD camera and a right-eye CCD camera at a fixed frequency on the left and right sides thereof. A mixed reality experience is provided by wearing liquid crystal shutter glasses that open and close.
An image system for supporting three-dimensional surveying using the mixed reality described in 3, 4, or 5. 7. A video display signal converted by the video conversion means is transmitted to a display unit of the personal computer and also to a display unit of a stereoscopic display, and two image splitters are arranged to separate left and right images. 7. The three-dimensional survey support image system using mixed reality according to claim 6, wherein the worker can perform stereoscopic viewing without glasses as the display. 8. The reference point automatic extraction means, based on the camera position and orientation data in the previous frame, in addition to the CCD camera principal point position and screen distance corrected in the initial image. Back-calculate the photo coordinates on the camera image of
A search range setting means for setting a predetermined search range around the coordinates and automatically extracting accurate photograph coordinates of the reference point based on the RGB information of the reference point therein. An image system for supporting three-dimensional surveying using the mixed reality described in 1, 2, 3, 4, or 5. 9. The reference point automatic extracting means includes a reference point marking means for marking in advance a color different from the surroundings of the reference point, and an operator selects the first frame of the initial image before executing the reference point automatic extraction. 9. The image system for supporting three-dimensional surveying using mixed reality according to claim 8, wherein the photograph coordinates of the reference point can be easily extracted. 10. The automatic reference point extraction means repeats the following four processes in automatic pattern recognition of RGB values within a search range when extracting a reference point. 3D survey support image system using mixed reality. R value of reference point i extracted in the previous frame (R
_i (t)), G value (G _i (t)), B value (B
_i (t)), width _(w i _(t)), stores the height _(h i _(t)); "truncation" _{(1) R i (t)} , G
A value (A means any one of R, G, and B) that takes the maximum value (A _i (t) _MAX ) of _i (t) and B _i (t)
Where A _i (t + 1) <A _i (t) _MAX / 2, and (2) a threshold dR (GB) for each R (GB) value
R (GB) _i (t) −dR (GB) <R (G
B) _i (t + 1) <R (GB) _i (t) + dR (GB)
If there are two or more sets of pixels in which the range of weight ≠ 0 is continuous in the “continuity processing” search range, the width w of the set is truncated.
_i (t + 1) and the height _hi (t + 1) are w
_i (t)) and only those close to _hi (t) are extracted; weighted barycentric coordinates for the A value within the search range are obtained and set as photograph coordinates.