KR101249369B1 - Apparatus and method for bundle adjustment of omni-directional images - Google Patents

Abstract

The present invention estimates the ground value and the external facial expression through the conjugation point using the luminous flux adjustment method using the omnidirectional image and the initial external facial expression acquired through the terrestrial mobile mapping system in which the omnidirectional camera and GPS / INS are integrated. The present invention uses the conjugate point coordinates of the omnidirectional image and the external facial expression elements of the omnidirectional camera as input values, and the calculated calculated values of the external facial expression elements and the ground point coordinates as output values. And three steps, including ground point estimation. The present invention can easily estimate the ground point and the external facial expressions by using the continuous omnidirectional camera image, and can improve the accuracy of the ground point coordinates.

Description

Apparatus and method for bundle adjustment of omni-directional images}

The present invention relates to an apparatus and method for determining absolute coordinates from an omnidirectional image. More particularly, the present invention relates to an apparatus and a method for determining absolute coordinates from an omnidirectional image through light beam adjustment.

Recently, attempts have been actively made to acquire spatial information by mounting omni-directional cameras in a mobile mapping system (MMS). The omnidirectional camera refers to a camera capable of acquiring image information in all directions from the photographing point because the field of view (FOV) is 360 °.

One example of the prior art is a technique for determining three-dimensional absolute coordinates through the omni-directional camera image. This technique addresses the equations and possibilities for determining three-dimensional absolute coordinates through omnidirectional cameras. This technique allows longer baselines than conventional frame cameras, resulting in highly accurate results. However, this technique is not suitable for digital photogrammetry using digital images because the accuracy of absolute coordinates is inferior to that of luminous flux adjustment.

As another example of the prior art, there is a technique of tracking and detecting the position of an object using an omnidirectional camera image. This technique uses a series of omnidirectional images to match and detect objects in the image. However, because this technique does not extract the conjugate points, the matching between ground and camera coordinate systems is not accurate.

The present invention has been made to solve the above problems, an omnidirectional image luminous flux for adjusting the omnidirectional image by using a mobile mapping system (MMS) incorporating an omnidirectional camera and a Global Positioning System / Inertial Navigation System (GPS / INS) An object of the present invention is to propose an adjustment device and method.

The present invention has been made to achieve the above object, an observation equation modeling unit for establishing an observation equation model using a collinear conditional expression for a conjugate point selected from consecutive omnidirectional images; A constraint model generator for generating constraint models in addition to the observation equation model by reflecting and not reflecting a predetermined constraint; And a luminous flux adjusting unit configured to bundle-adjust the omnidirectional image by using the generated constraint equation models and the observation equation model established.

Preferably, the luminous flux adjusting unit estimates a ground point and an external facial expression for extracting a three-dimensional model of an object as related to the omnidirectional image through the luminous flux adjustment. More preferably, the observation equation modeling unit establishes an observation equation model using the conjugate point coordinates of the omnidirectional image as the observation data to be input and the ground point and the external expression element as an unknown variable to be output.

Preferably, the constraint model generation unit uses an external expression element pre-stored as the constraint and a ground reference point positioned on the ground coordinate system. More preferably, the omnidirectional image luminous flux adjusting device uses the satellite navigation device (GPS) or the inertial navigation device (INS) to obtain the position information and the posture information of the omnidirectional image acquisition device that acquires the omnidirectional image with the external facial expression element. Obtaining position / posture information acquisition unit; Alternatively, the apparatus may further include a ground reference point acquisition unit for acquiring the ground reference point using the satellite navigation apparatus or an optical wave, and further comprising a position / posture information acquisition unit, and may further include an omnidirectional image acquisition unit for acquiring the omnidirectional image. .

Preferably, the observation equation modeling unit measures an image point coordinate of an image coordinate system related to the omnidirectional image, a ground point coordinate located on a ground coordinate system, and an image point corresponding to a ground point located on the image coordinate system. The observation equation model is established using a measurement error generated at the time and a coordinate transformation coefficient between the ground coordinate system and the image coordinate system. More preferably, the observation equation modeling unit, a linearization unit for linearizing the equation by the image point coordinates, the ground point coordinates, the measurement error, and the coordinate transformation coefficients using a Taylor series; Substituent for substituting the linearized equation in the form of Jacobian; A derivative unit for differentiating a substituted expression for each predetermined component; And a sizing unit for establishing the observation equation model using differential equations differentiated by the derivative by using the rotation matrix of the rotation angle of each omnidirectional image as a rotation matrix for transforming between the ground coordinate system and the image coordinate system. Include.

In addition, the present invention provides an observation equation modeling step of establishing an observation equation model using a collinear conditional expression for a conjugate point selected from consecutive omnidirectional images; A constraint model generation step of generating constraint model in addition to the observation equation model by reflecting and not reflecting a predetermined constraint; And a luminous flux adjustment step of bundle-adjusting the omnidirectional image by using the generated constraint equation models and the observation equation model established.

Preferably, the luminous flux adjustment step estimates a ground point and an external facial expression for extracting a three-dimensional model of an object as related to the omnidirectional image through the luminous flux adjustment. More preferably, the step of establishing an observation equation model establishes an observation equation model using the conjugate point coordinates of the omnidirectional image as the observation data to be input and the ground point and the external expression element as an unknown variable to be output.

Preferably, the constraint model generation step uses an external expression element previously stored as the constraint and a ground reference point positioned on the ground coordinate system. More preferably, the omnidirectional image luminous flux adjusting method may include position information and posture information of the omnidirectional image acquisition device that acquires the omnidirectional image with the external facial expression element using a satellite navigation device (GPS) or an inertial navigation device (INS). Acquiring position / posture information; Alternatively, the method may further include a ground reference point obtaining step of obtaining the ground reference point using the satellite navigation apparatus or an optical wave, and further including the observation data obtaining step of obtaining the observation data using a ground photogrammetry system (MMS). have.

Preferably, the step of establishing the observation equation model measures an image point coordinate of an image coordinate system related to the omnidirectional image, a ground point coordinate located on a ground coordinate system, and an image point corresponding to a ground point located on the image coordinate system. The observation equation model is established using the measurement error generated at the time and the coordinate transformation coefficient between the ground coordinate system and the image coordinate system. More preferably, the step of establishing the observation equation model, the linearization step of linearizing the equation by the image point coordinates, the ground point coordinates, the measurement error, and the coordinate transformation coefficients using a Taylor series; A substitution step of replacing the linearized expression in the form of a Jacobi matrix; A derivative step of differentiating the substituted equation for each predetermined component; And a step of establishing the observation equation model using differential equations by the differential step using the rotation matrix by the rotation angle of each axis of the omnidirectional image as a rotation matrix for transforming between the ground coordinate system and the image coordinate system. It includes.

According to the present invention, the following effects can be obtained by adjusting the luminous flux of an omnidirectional image using a mobile mapping system in which an omnidirectional camera and GPS / INS are integrated. First, by establishing a collinear equation suitable for the omnidirectional image and an observation equation based on this equation, it is possible to easily estimate the ground point and the external facial expressions by using the continuous omnidirectional camera image. Second, by establishing and verifying various types of probabilistic constraint expressions using ground reference points acquired through external facial expression elements obtained from GPS / INS, stationary GPS, and total stations, accuracy of ground point coordinates can be improved.

1 is a block diagram schematically illustrating an internal configuration of an omnidirectional image luminous flux adjusting apparatus according to a preferred embodiment of the present invention.
2 is a block diagram showing in detail the internal configuration of the omnidirectional image light beam adjusting apparatus according to a preferred embodiment of the present invention.
3 is a schematic diagram of a methodology according to the present embodiment.
4 is a diagram illustrating a relationship between a ground point and an image point based on a central projection principle.
5 is a reference diagram for explaining a methodology according to the present embodiment.
6 to 13 are reference diagrams for explaining an experiment with respect to the present embodiment.
14 is a flowchart illustrating a method for adjusting the omnidirectional image luminous flux according to an exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the preferred embodiments of the present invention will be described below, but it is needless to say that the technical idea of the present invention is not limited thereto and can be variously modified by those skilled in the art.

1 is a block diagram schematically illustrating an internal configuration of an omnidirectional image luminous flux adjusting apparatus according to a preferred embodiment of the present invention. 2 is a block diagram showing in detail the internal configuration of the omnidirectional image light beam adjusting apparatus according to a preferred embodiment of the present invention. The following description refers to FIGS. 1 and 2.

According to FIG. 1, the omnidirectional image luminous flux adjusting apparatus 100 includes an observation equation modeling unit 110, a constraint equation model generation unit 120, a luminous flux adjusting unit 130, a power supply unit 140, and a main control unit 150. do.

The observation equation modeling unit 110 performs a function of establishing an observation equation model using a collinearity equation for a conjugate point selected from consecutive omnidirectional images. Observation equations based on collinear conditions can be established with image coordinate values of conjugate points obtained from two or more omnidirectional images. The observation equation modeling unit 110 establishes an observation equation model that uses the coordinates of the coordinates of the omnidirectional image as the observation data to be input and uses the ground point and the external facial expression as an unknown variable to be output.

Observation equations consist of observations, unknowns, and mathematical models between observations and unknowns. Observation values mean image coordinate values and observation errors acquired in the image coordinate system. The unknown means the ground point expressed in the ground coordinate system and the external expression elements of the camera. The mathematical model consists of collinear conditions and coordinate transformations. The observation equation model is constructed using collinear equations, observations and observation errors, and unknowns. The collinear equations can be established with coordinate transformation coefficients (ground points on the ground coordinate system to ground points on the image coordinate system), collinear conditions under which the ground points are projected onto the image. There are two components of the coordinate transformation coefficient: the rotation transformation coefficient and the homing transformation coefficient. In this case, the observation equation modeling unit 110 is an image point coordinate of an image coordinate system (camera coordinate system) related to the omnidirectional image, a ground point coordinate located on the ground coordinate system, and an image point corresponding to the ground point located on the image coordinate system. The observation equation model is established by using the measurement error that occurs when measuring and the coordinate transformation coefficient between the ground coordinate system and the image coordinate system. In view of this, the observation equation modeling unit 110 includes a linearization unit 111, a substitution unit 112, a differential unit 113, and a determination unit 114 as shown in FIG. 2B. can do.

The linearization unit 111 performs a function of linearizing an equation based on image point coordinates, ground point coordinates, measurement error, and coordinate transformation coefficients using the Taylor series. The substitution unit 112 performs a function of replacing the linearized expression in the form of Jacobian matrix. The derivative unit 113 performs a function of differentiating the substituted expression for each predetermined component. The derivative unit 113 uses an unknown variable as a predetermined component. In the present embodiment, the unknown variable includes an external facial expression element and a ground point coordinate, and in particular, the external facial expression element includes a positional element and a posture element. The sizing unit 114 performs a function of establishing an observation equation model using differential equations differentiated by a derivative using a rotation matrix of rotation angles of the omnidirectional image axes as a rotation matrix for transforming the ground coordinate system and the image coordinate system. do.

The constraint model generation unit 120 reflects and not reflects a predetermined constraint to generate the constraint models in addition to the observation equation model.

The constraint model generation unit 120 may use an external expression element previously stored as a constraint and a ground reference point positioned on the ground coordinate system. In consideration of this point, the omnidirectional image luminous flux adjusting apparatus 100 includes a position / posture information obtaining unit 210, a ground reference point obtaining unit 220, and an omnidirectional image obtaining unit 230 as shown in FIG. 2A. It may further include at least one component.

The position / posture information acquisition unit 210 acquires the position information and the attitude information of the omnidirectional image acquisition device that acquires the omnidirectional image using an external facial expression element using the GPS or the inertial navigation device (INS). To perform. The ground reference point acquisition unit 220 performs a function of acquiring the ground reference point using GPS / INS, an optical wave. The omnidirectional image acquisition unit 230 includes a position / posture information acquisition unit 210 and performs a function of acquiring an omnidirectional image.

The luminous flux adjusting unit 130 performs a bundle adjustment of the omnidirectional image by using the generated constraint equation models and the observation equation model established. The light beam controller 130 estimates the ground point and the external facial expression for extracting the 3D model of the object as related to the omnidirectional image by adjusting the light beam.

The power supply unit 140 performs a function of supplying power to each unit constituting the omnidirectional image luminous flux adjusting device 100.

The main control unit 150 performs a function of controlling the overall operation of each unit constituting the omnidirectional image luminous flux adjusting device 100.

Next, the omnidirectional image luminous flux adjusting device 100 will be described with reference to one embodiment. In the following embodiment, a methodology for estimating the ground value and the external facial expression through the conjugation point using the luminous flux adjustment method using the omnidirectional image and the initial external facial expression obtained through the terrestrial mobile mapping system in which the omnidirectional camera and GPS / INS are integrated is described. Introduce.

In the light beam adjustment method using the omnidirectional image, the method proposed in this embodiment, the conjugate point coordinates of the omnidirectional image and the external facial expression elements of the omnidirectional camera are input values, and the calculated calculated values of the external facial expression elements and the ground point coordinates are output values. 3 shows an overview of the methodology using an omnidirectional image. As shown in FIG. 3, the proposed method is largely composed of three steps: establishing a new observation equation, establishing a stochastic constraint, and estimating external facial expressions and ground points.

는 각각 외부 표정 요소와 지상점 좌표 미지수이며, e_y는 오차를 의미한다.The first step is to construct a new observation equation model using the collinear conditional equations suitable for the omnidirectional image. In this embodiment, this step may be performed by the observation equation modeling unit of FIG. 1. According to the principle of central projection, a straight line connecting the center of the camera projection and the ground point meets at one point with a sphere centered on the camera projection center. Then, we establish an observation equation with the conjugate point and the external expression element as observations and the unknown expression element and the ground point coordinates as unknowns. Equation 1 below shows the observation equation. In this case, Y is a conjugate point coordinate value on the omnidirectional image.

Are the external facial expressions and the ground coordinate coordinates, respectively, and e _y represents the error.

The next step is to establish stochastic constraints with the external facial expressions of cameras acquired from GPS / INS mounted on the ground MMS and the ground reference points acquired through stationary GPS and total station. In this embodiment, this step may be performed by the constraint model generator of FIG. 1. Equations 2 and 3 are probability constraint expressions for the external facial expression element and the ground control point, respectively. Z ₁ and Z ₂ are the observation vectors for the external facial expression and ground control points, respectively. In this step, a total of eight mathematical models are completed by combining the unknowns estimated in the previous step and whether two kinds of probability constraints are applied.

The last step is to adjust the external facial expressions and ground point coordinates of the image and analyze the results by applying the real data acquired through the ground MMS to the luminous flux adjustment method according to the various mathematical models proposed above. In this embodiment, this step may be performed by the luminous flux adjusting part of FIG. Accuracy verification is confirmed by errors between checkpoints measured by stationary GPS and total stations and ground points estimated by the proposed methodology.

Hereinafter, each step will be described in more detail.

end. Establish Observation Equation

In a general frame camera, an image point is generated at a point where a ground point in a three-dimensional space meets a two-dimensional focal plane through the main point of the lens. In the case of omnidirectional cameras, however, image points are created on the surface of the three-dimensional sphere rather than the two-dimensional focal plane. Due to the difference between the frame camera and the omni-directional camera, the collinear conditional formula suitable for the omnidirectional image was established.

Equation 4 converts a ground point vector on a ground coordinate system into a ground point vector on camera coordinates. ^C P is the ground point vector on the camera coordinate system and ^G P is the ground point vector on the ground point coordinate system. ^C _G R is a rotational conversion formula between the ground coordinate system and the camera coordinate system, and ^G O _C is the center coordinate of the camera coordinate system on the ground coordinate system.

로 수학식 5와 같이 표현이 가능하다. ^CP_x, ^CP_y, ^CP_z는 ^CP의 좌표를 의미한다.The origin of the camera coordinate system corresponds to the projection center ^G O _C of the camera, and the image point ρ exists on a spherical surface centered on the camera projection center. According to the central projection principle, the center of the projection, the image point, and the ground point ^C P exist on one straight line as shown in FIG. 4. The image point can be represented by the horizontal angle α and the vertical angle β of the polar coordinate system.

It can be expressed as shown in Equation 5. ^C P _x , ^C P _y , and ^C P _z refer to the coordinates of ^C P.

Equation 4 and Equation 5 can be used to express the relationship between the image point and the ground point in the omnidirectional image and to derive a collinear equation suitable for the omnidirectional image. Using this collinear equation, an observation equation suitable for the omnidirectional image can be established as shown in Equation 6. Where σ ₀ ² is a measurement error that can occur when measuring an image point and I ₂ is a unit matrix of 2 × 2. Equation 6 represents the rotational transformation between ground coordinates ^C P _x , ^C P _y , ^C P _z and projection center coordinates X _C , Y _C , Z _C and the ground coordinates and the camera coordinate system. It contains a total of nine coefficients.

는 앞서 언급한 9개의 계수에 대한 미지수이고

은

의 초기 근사값이며 f는 공선 조건식의 함수이다.Using the Taylor series of Equation 6 can be linearized as shown in Equation 7.

Is the unknown for the nine coefficients mentioned above

silver

Is an initial approximation of f and f is a function of the collinear condition.

Equation (8) is the result of expressing the equation (7) in a Jacobian matrix. t ₁ and t ₂ were substituted as in Equation (9).

Differentiation is performed by dividing Equation 8 by components such as Equation 10, Equation 11, and Equation 12, respectively.

에 대해 미분한 수식이고 수학식 13은 각 미지수별 성분을 보여주고 있다.

는 외부 표정 요소 중 위치 요소,

는 외부 표정 요소 중 자세 요소,

는 지상점의 3차원 좌표를 의미한다.Equation 12 is ^C P

Equation 13 is a differential equation for and Equation 13 shows the components for each unknown.

Is the location element,

The posture element,

Is the three-dimensional coordinates of the ground point.

및

에 대한 미분값은 비교적 단순한 형태로 각각 수학식 14와 수학식 15처럼 표현된다. ^C P

And

The derivative value for is expressed in a relatively simple form as shown in Equations 14 and 15, respectively.

에 대한 미분값은 수학식 16과 같으며, ^C _GR는 각 축의 회전각을 곱한 회전 행렬로서 수학식 17과 수학식 18로 표현할 수 있다. 수학식 17은 수학식 14 내지 수학식 16의 ^C _GR을 정의하는 것이다. ^C _GR은 카메라 좌표계에서 지상 좌표계로 변환할 때 쓰이는 회전 행렬이다. 수학식 18은 수학식 17에서 정의하고 있는 R_x(ω), R_y(φ), R_z(κ)를 정의하는 식이다. 여기서, ω, φ, κ는 각각 X, Y, Z축의 회전각이다.Like ^C P

The derivative value for is equal to Equation 16, and ^C _G R may be expressed as Equation 17 and Equation 18 as a rotation matrix multiplied by the rotation angle of each axis. Equation 17 defines ^C _G R of Equations 14 to 16. ^C _G R is a rotation matrix used to convert from the camera coordinate system to the ground coordinate system. Equation 18 is an equation for defining R _x (ω), R _y (φ), and R _z (κ) defined in Equation 17. Here, ω, φ, and κ are rotation angles of the X, Y, and Z axes, respectively.

Thus, the derivative values for ω, φ, and κ of ^C _G R are expressed as in Equations 19, 20, and 21, respectively.

Equations 14 to 21 described above solve the observation equation and correspond to y = A _e A _p ξ _e + e _y in Equation 22 to be described later.

I. Establish constraint by type

Equation 22 shows an observation equation using an external facial expression element and a ground control point as constraints. ξ _e is unknown for external facial expressions and ξ _p is unknown for ground control points. y is the observed value of the conjugate point, and A _e , A _p are the design mattresses derived from the collinear conditional equation that differentiates the unknown ξ _e of the external facial expression element and the unknown ξ _p of the ground reference point. z ₁ is the observation vector of the external facial expression element measured from GPS / INS and z ₂ is the observation vector to ground reference point measured from stationary GPS and total station. K ₁ , K ₂ are design mattresses for external facial expression factors and ground control point constraints. e _y , e _z1 , e _z2 And the like mean an error vector associated with an observation vector such as y, z1, z2, and the like. σ ₀ ² is the variance of the unknown component, P _y ^{- 1} means a cofactor for the mattress e _y. ^-1 P _z1 is the cofactor of the mattress e _z1 indicating the accuracy of the GPS / INS, P _z2 ^{- 1} means a cofactor of the mattress e _z2 indicating the accuracy of the ground control points.

In the present invention, a total of four types were defined according to the applied constraints, and the accuracy results for each type were analyzed. In Type 1, the ground point is estimated using the initial external facial expression elements obtained from GPS / INS without using constraints. In type 2 and type 3, external facial expression elements and ground control points were applied to constraint equations, respectively. In Type 4, both external facial expression elements and ground control points were applied to the constraint. Figure 5 (a) shows the elements and constraints used for each type, Figure 5 (b) shows the mathematical model used for each type.

All. Experimental Results and Analysis

The omnidirectional image and position / posture data used in the experiments of the present invention were obtained through a terrestrial MMS equipped with an omnidirectional camera and GPS / INS. The omnidirectional camera is Point Gray Research's Ladybug3 model, which integrates images acquired through six individual CCDs into a single image to generate an omnidirectional image and has a field of view of 360 degrees in the horizontal direction and 180 degrees in the vertical direction. Figure 6 (a) shows the specification of Ladybug3 and Figure 6 (b) shows its appearance.

A process in which the omnidirectional camera acquires six individual images and finally generates one integrated image is illustrated in FIG. 7A. FIG. 7A illustrates a generation process of the omnidirectional camera image (Point Gray Research, 2008). In the first step, Bayer-tiled Raw images obtained through individual CCDs are compressed to JPEG and then transferred to a computer. The computer decompresses the image and converts it to an RGB color image for transmission to the graphics card. Here, a final image is generated through a rectification process of removing various distortions, a projection process of converting a coordinate system of an image texture, and a blending process of connecting images.

GPS / INS was mounted on the terrestrial MMS to acquire the external facial expression of the moment when the omnidirectional image was acquired. The onboard system is an Applanix POSLV 420PP model that is suitable for mobile systems such as mobile mapping systems (Applanix, 2009). It consists of an inertial observation device (IMU) with an accelerometer that measures acceleration in three axes and a gyroscope that measures the angular velocity of three rotational axes. GPS uses the GMS (GNSS Azimuth Measurement Subsystem), which not only measures the position of three-dimensional absolute coordinates but also corrects the error of the INS using two GPS antennas. It is a system that measures the relative position vector between two GPS antennas using the carrier of GPS and continuously corrects the error of INS. Figure 7 (b) shows the accuracy of the GPS / INS system mounted.

Figure 8 (a) shows the appearance of the ground MMS used in this experiment. At the top of the system is an omni-directional camera, below it an IMU, GPS antenna, and receiver.

In this experiment, 24 consecutive omnidirectional images of the road around Osan City Hall were used. The distance between the first image and the last image is about 100m and the distance between two consecutive images is about 4m. 8 (b) shows a place where the present experimental data was obtained, that is, a test subject. The yellow dashed line across the middle of the figure indicates the direction of movement of the vehicle. The numbers in yellow are ground points obtained from stationary GPS and total stations, some of which were used as reference points and the other ground points were used as check points.

9 (a) shows the positions of the ground reference point and the inspection point, and shows the positions and indexes of the ground points described above. The red index means this ground point is used as a reference point, and the black index is used as a check point. And conjugation point was additionally selected to connect the continuous images. 9 (b) is an example showing the position of a pair of conjugate points.

There are a total of 28 ground points associated with all conjugate points used in this experiment. Each ground point was taken from a minimum of four images to a maximum of eleven images. The average number of ground points included in two consecutive chapters at the same time is 7.9, which is more than 5, the minimum number of conjugate points required for mutual expression. Information about the image and the ground point used in the experiment is described in Figure 10 (b). 10 (b) shows experimental data information. The ground point included in each image can be confirmed through (a) of FIG. 10. 10A illustrates a relationship between an image and a ground point. The row represents the index of the ground point, and the column represents the index of the image. If there is a ground point in the video, it will be displayed in red. For example, ground point 1 is included in image 1 but not image 2.

And, the measurement error when measuring the conjugate point through the omnidirectional image was set to ± 1 °. The measurement error of ground reference point and GPS is ± 5 cm and that of INS is ± 0.05 °.

The experiment was carried out using the omnidirectional image and GPS / INS data obtained from the test area and the ground reference point / test point measured by the stationary GPS and total station. Based on the eight mathematical models proposed above, the results can be confirmed through (a) of FIG. FIG. 11A illustrates the ground point error estimated for Types 1 and 2. FIG. In FIG. 11, it can be seen that the RMSE difference between the type B, which applies the external facial expression element as the probability constraint, and the RMSE difference between the type A, which does not apply the probability constraint at all, are very small. This means that the estimated external facial expression factor applied as a probability constraint in Type B is almost the same as the initial external facial expression factor obtained from GPS / INS of terrestrial MMS. This is because the measurement error when measuring the conjugate point in the omnidirectional image is larger than the GPS / INS measurement error. Therefore, the mutual expression process performed through the conjugate point connecting the consecutive images cannot improve the accuracy of the external expression elements.

11 (b) shows the error of the ground point estimated through the type 3 and type 4. In Type 3, only four ground control points (Nos. 1, 4, 8, and 10) were applied to the probability constraint. Comparing the RMSE with other types, the error is very low. Through this, it was found that the ground point can be estimated with the accuracy of ± 5cm using a small number of ground reference points. However, to use Type 3, the measurement of the ground reference point with high accuracy must be preceded. In Type 4, all four ground control points and external facial expressions are applied to the probability constraint condition. The RMSE is improved compared to Types 1 and 2, but did not show good results compared to Type 3.

FIG. 12A shows the residual between the measured image point coordinates and the adjusted calculated image point coordinates for each type. The large residual means that the difference between the measured image point coordinates and the adjusted calculated image point coordinates is large and that the estimation result of the external facial expression factor and the ground point coordinates is not good. The amount of residual for each type is the same as in FIG. 12A, and the statistic can be confirmed through FIG. 12B. 12 (a) to (b), it can be seen that Type 3 has a very low residual compared to other types, which means that the accuracy of the mathematical model is very high.

FIG. 13 shows the ground points used for the estimated ground points, external facial expression factors, and probability constraints and the check points used for accuracy comparison. Blue points are estimated ground points, and red points are estimated external facial expressions. The red circle is the ground reference point and the green circle is the checkpoint.

Next, the adjustment method by the omnidirectional video luminous flux adjustment apparatus of FIG. 1 is demonstrated. 14 is a flowchart illustrating a method for adjusting the omnidirectional image luminous flux according to an exemplary embodiment of the present invention. The following description refers to FIG. 14.

First, an observation equation model is established using a collinear conditional expression for a conjugate point selected from successive omnidirectional images (observation equation model establishment step, S10).

In the observation equation model establishment step (S10), an image point coordinate of an image coordinate system related to an omnidirectional image, a ground point coordinate located on the ground coordinate system, and an image point corresponding to the ground point generated on the image coordinate system are generated. The observation equation model can be established using the measurement error and the coordinate transformation coefficient between the ground coordinate system and the image coordinate system. In view of this point, the observation equation model establishment step S10 may include a linearization step, a substitution step, a derivative step, and a formulation step.

The linearization step refers to a step of linearizing an equation based on image point coordinates, ground point coordinates, measurement errors, and coordinate transformation coefficients using a Taylor series. The substitution step refers to the step of replacing the linearized expression in the form of Jacobian matrix. The derivative step refers to a step of differentiating a substituted equation for each predetermined component. The establishing step refers to a step of establishing an observation equation model using equations differentiated by a differential step using a rotation matrix of rotation angles of each of the omnidirectional images as a rotation matrix for transformation between the ground coordinate system and the image coordinate system.

In the step of establishing the observation equation model (S10), the observation equation model may be established by using the coordinates of the coordinates of the omnidirectional image as the observation data to be input, and the ground point and the external facial expression as an unknown variable to be output.

After the observation equation model establishment step S10, the constraint equation models are generated in addition to the observation equation model by reflecting and not reflecting a predetermined constraint (constraint model generation step, S20). In the constraint model generation step (S20), an external expression element previously stored as a constraint and a ground reference point positioned on the ground coordinate system may be used. In consideration of this point, the omnidirectional image luminous flux adjusting method according to the present embodiment may further perform at least one of position / posture information acquiring, ground reference point acquiring, and observation data acquiring.

In the step of obtaining position / posture information, position information and attitude information of an omnidirectional image acquisition device which acquires an omnidirectional image using an external facial expression element using a satellite navigation device (GPS) or an inertial navigation device (INS). In the ground reference point acquisition step, a ground reference point is obtained by using a satellite navigation device or an optical wave. The position / posture information acquisition step and the ground reference point acquisition step are performed between the observation equation model establishment step S10 and the constraint equation model generation step S20. However, the present invention is not limited thereto, and may be performed before the observation equation modeling step S10. In the observation data acquisition step, observation data is acquired using a ground photogrammetry system (MMS). The observation data refers to location / posture information, ground reference point, and omnidirectional image of the image acquisition device using GPS / INS. In the present embodiment, it is also possible to acquire only the omnidirectional image in the observation data acquisition step. The observation data acquisition step is performed between the constraint model generation step S20 and the luminous flux adjustment step S30. However, the present invention is not limited thereto, and may be performed between the observation equation model establishment step S10 and the constraint equation model generation step S20, or may be performed before the observation equation model establishment step S10.

After the constraint model generation step S20, the omnidirectional image is bundle adjusted using the generated constraint equation models and the observation equation model established (S30). In the luminous flux adjustment step (S30), the ground point and the external facial expression for extracting the three-dimensional model of the object as related to the omnidirectional image through the luminous flux adjustment may be estimated.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

In the present invention, it can be seen that the ground point and the external facial expressions can be estimated using the continuous omnidirectional camera image. For this, a new collinear equation suitable for the omnidirectional image was established, and an observation equation was also established. In addition, four types of accuracy were verified by establishing probability constraint equations using external facial expressions acquired from GPS / INS and ground reference points acquired through stationary GPS and total stations. Through the present invention, if the ground reference point with a high accuracy it can be seen that the ground point extraction with an accuracy of about 5cm. This means that it can be widely used in fields such as precise three-dimensional urban modeling. The present invention is also applicable to the field of real time aerial data acquisition system.

100: omnidirectional image luminous flux adjusting device 110: observation equation model formulation
111: linearization unit 112: substitution unit
113: differential part 114: upsetting part
120: constraint model generation unit 130: luminous flux adjusting unit
210: location / attitude information acquisition unit 220: ground reference point acquisition unit
230: omnidirectional image acquisition unit 240: collinear conditional setting unit
241 conversion unit 242 calculation unit
243: setting unit

Claims (14)

An observation equation modeling unit for establishing an observation equation model using a collinear condition equation for a conjugate point selected from consecutive omnidirectional images;
A constraint model generator for generating constraint models by adding predetermined constraints to the observation equation model; And
Luminous flux adjusting unit for bundling the omnidirectional image using the generated constraint equation models and the observation equation model established
The omnidirectional image beam adjustment device comprising a. The method of claim 1,
And the luminous flux adjusting unit estimates a ground point and an external facial expression for extracting a three-dimensional model of an object as related to the omnidirectional image through the luminous flux adjustment. The method of claim 1,
And the constraint model generator uses an external facial expression element pre-stored as the constraint and a ground reference point positioned on a ground coordinate system. The method of claim 3, wherein
A position / posture information acquisition unit for acquiring position information and attitude information of the omnidirectional image acquisition device which acquires the omnidirectional image with the external facial expression element using a satellite navigation device (GPS) or an inertial navigation device (INS); And
And a ground reference point obtaining unit which obtains the ground reference point using the satellite navigation apparatus or the optical wave.
The location / posture information acquisition unit,
And a omnidirectional image obtaining unit for acquiring the omnidirectional image. The method of claim 1,
The observation equation modeling unit is a measurement generated when measuring an image point coordinate of an image coordinate system related to the omnidirectional image, a ground point coordinate located on a ground coordinate system, and an image point corresponding to a ground point located on the image coordinate system. And the observation equation model is established using an error and a coordinate transformation coefficient between the ground coordinate system and the image coordinate system. The method of claim 5, wherein
The observation equation model formulation,
A linearizer which linearizes an expression by the image point coordinates, the ground point coordinates, the measurement error, and the coordinate transformation coefficients using a Taylor series;
Substituent for substituting the linearized equation in the form of Jacobian;
A derivative unit for differentiating a substituted expression for each predetermined component; And
Sizing unit for establishing the observation equation model using differential equations differentiated by the differentiation unit using the rotational matrix by the rotation angle of each omnidirectional image as a rotation matrix for transforming between the ground coordinate system and the image coordinate system
The omnidirectional image beam adjustment device comprising a. The method of claim 2,
The observation equation modeling unit establishes an observation equation model using the conjugate point coordinates of the omnidirectional image as the observation data to be input, and the ground point and the external facial expression element as an unknown variable to be output. . An observation equation modeling step of establishing an observation equation model using a collinear condition equation for a conjugate point selected from successive omnidirectional images;
Generating a constraint equation model by adding a predetermined constraint to the observation equation model; And
Luminous flux adjustment step of bundle adjustment of the omnidirectional image using the generated constraint equation models and the observation equation model established
The omnidirectional image beam adjustment method comprising a. The method of claim 8,
And adjusting the luminous flux by estimating a ground point and an external facial expression for extracting a three-dimensional model of an object related to the omnidirectional image through the luminous flux adjustment. The method of claim 8,
And generating the constraint model using an external expression element previously stored as the constraint and a ground reference point positioned on a ground coordinate system. 11. The method of claim 10,
A position / posture information acquiring step of acquiring position information and attitude information of the omnidirectional image acquisition device which acquires the omnidirectional image with the external facial expression element using a satellite navigation device (GPS) or an inertial navigation device (INS);
A ground reference point obtaining step of obtaining the ground reference point using the satellite navigation apparatus or the optical wave; And
Observation data acquisition step of acquiring observation data using a ground photogrammetry system (MMS)
The omnidirectional image luminous flux adjustment method further comprising. The method of claim 8,
The step of establishing an observation equation model is generated when measuring an image point coordinate of an image coordinate system related to the omnidirectional image, a ground point coordinate located on a ground coordinate system, and an image point corresponding to a ground point located on the image coordinate system. And the observation equation model is established using a measurement error and a coordinate transformation coefficient between the terrestrial coordinate system and the image coordinate system. 13. The method of claim 12,
The observation equation model formulation step,
Linearizing a calculation equation based on the image point coordinates, the ground point coordinates, the measurement error, and the coordinate transformation coefficients using a Taylor series;
A substitution step of replacing the linearized expression in the form of a Jacobi matrix;
A derivative step of differentiating the substituted equation for each predetermined component; And
Formulation step of establishing the observation equation model with the equations differentiated by the derivative step using the rotation matrix by the rotation angle of each axis of the omnidirectional image as a rotation matrix for the transformation between the ground coordinate system and the image coordinate system
The omnidirectional image beam adjustment method comprising a. The method of claim 9,
In the step of establishing the observation equation model, the observation image model is adjusted by using the coordinates of the omnidirectional image as the observation data to be input and establishing the observation equation model using the ground point and the external facial expression as an unknown variable. Way.

Images