KR20220074103A - Device and method of image registration for 3d models - Google Patents

Abstract

According to the present invention, an image registration method for a 3D model is performed in an image registration apparatus for a 3D model.
Here, the image registration method for the three-dimensional model may include: implementing a three-dimensional model of a facility in a virtual environment; generating a virtual image of the 3D model through virtual imaging in the virtual environment; calculating an error between the virtual image and the captured image obtained by photographing the actual facility by the photographing apparatus; and matching the virtual image and the captured image by correcting the calculated error.
According to the present invention, by matching an image captured by a camera with a three-dimensional model of an object to be photographed implemented in a virtual environment, it is possible to accurately check and diagnose an object to be photographed even through the three-dimensional model.

Description

DEVICE AND METHOD OF IMAGE REGISTRATION FOR 3D MODELS

The present invention relates to an image matching apparatus and method for a three-dimensional model, and more particularly, by matching an image captured by a camera with a three-dimensional model of a photographing object implemented in a virtual environment, It relates to an image matching apparatus and method for a 3D model capable of accurately checking and diagnosing .

As one of the methods for facility management including safety diagnosis and damage level inspection, there is a method of obtaining and utilizing information related to the condition and location of a facility by using an imaging device. When the information obtained through the imaging equipment is applied to a three-dimensional model of a facility to be managed implemented in a virtual environment, etc., there is an advantage in that the facility to be managed can be easily inspected in a remote location.

On the other hand, by using an unmanned aerial vehicle in a situation or place where access by manpower is not easy, a detailed inspection of the facility can be performed. The inspection and management method of facilities using such an unmanned aerial vehicle must be able to accurately identify the location of some of the facilities displayed in the image captured by the unmanned aerial vehicle, and the technology that can accurately identify minute damage to the facility is required. will be needed

Specifically, in order to obtain an image from a photographing device mounted on an unmanned aerial vehicle and to extract quantitative information about a facility from such an image, the location at which the image was captured and the posture of the camera at the time of photographing must be determined in three-dimensional space. Accordingly, as a device for measuring the position and attitude of an unmanned aerial vehicle (or a photographing device mounted thereon), a GNSS (Global Navigation Satellite System) signal receiving device and IMU (Inertial Measurement Unit; Inertial Measurement Unit) are used. may be used, and according to convenience, a GNSS/IMU in which the functions of the two devices are merged into one may be used.

However, in a situation where the GNSS signal is not received enough to determine the exact position of the unmanned aerial vehicle, it may be difficult to determine the position and posture of the acquired image. For example, if an unmanned aerial vehicle is moved to the lower part of the bridge facility for a detailed inspection of the lower part of the bridge facility, the reception of the GNSS signal may not be smooth. Since this does not match exactly, it may be difficult to accurately match the image acquired to the facility implemented with the aforementioned three-dimensional model.

In addition, there is a problem that, in addition to the problems related to GNSS signal reception, there may be difficulties in matching the acquired images even if the camera alignment state of the imaging equipment is not accurately determined or the shape of the implemented 3D model is not accurate. have.

Korean Patent Registration No. 10-2038127

The present invention is devised to solve the above problems, and a process of calculating an error between the two images by comparing a captured image obtained through a photographing device and a virtual image generated for a 3D model, and correcting the calculated error Accurate matching can be achieved through

According to an aspect of the present invention, the method of matching an image to a 3D model is performed by an apparatus for matching an image to a 3D model. Here, the image registration method for the three-dimensional model may include: implementing a three-dimensional model of a facility in a virtual environment; generating a virtual image of the 3D model through virtual imaging in the virtual environment; calculating an error between the generated virtual image and a photographed image obtained by photographing an actual facility by the photographing apparatus; and matching the virtual image and the captured image by correcting the calculated error.

In one embodiment, generating a virtual image of the 3D model; collecting the photographing position and photographing posture information for each of the photographed images acquired by the photographing apparatus; and deriving a photographing position and photographing posture in the virtual environment corresponding to the collected photographing position and photographing posture information, and generating a virtual image for the three-dimensional model based on the derived photographing position and photographing posture. ; may be included.

In an embodiment, the calculating of the error may include extracting an outline of a facility included in the virtual image and the captured image.

In an embodiment, the matching of the virtual image and the captured image may include matching the virtual image and the captured image based on a projection transformation method using the extracted outline.

In an embodiment, the calculating of the error may include: matching images determined to have the same photographing position and photographing posture among the virtual image and the photographed image; and overlapping the matched images to be projected on each other.

In an embodiment, the matching of the virtual image and the captured image may include matching the virtual image and the captured image by applying the projection transformation method to each of the overlapping images.

In an embodiment, matching the virtual image and the captured image based on a projection transformation method may include calculating transformation coordinates in the virtual environment with respect to the acquired coordinates of the captured image.

According to another aspect of the present invention, an apparatus for registering an image for a 3D model performs an image matching method for a 3D model. Here, the image matching apparatus for the three-dimensional model may include: a three-dimensional model generator for implementing a three-dimensional model of a facility in a virtual environment; a virtual image generating unit generating a virtual image of the 3D model through virtual imaging in the virtual environment; an error calculation unit for calculating an error between the generated virtual image and a captured image obtained by photographing an actual facility by the photographing apparatus; and an image matching unit configured to match the virtual image and the captured image by correcting the calculated error.

An apparatus and method for matching an image for a three-dimensional model according to an embodiment of the present invention accurately detects the occurrence of a discrepancy between a virtual image of an implemented three-dimensional model and a captured image to be matched, and corrects it to correct a three-dimensional (3D) almost identical to an actual facility There is an effect that the model can be implemented.

Since the image registration apparatus and method for a three-dimensional model according to an embodiment of the present invention can accurately match a photographed image to a three-dimensional model for a facility, it is possible to easily and precisely check a facility to be managed even at a remote location. have.

Since the image registration apparatus and method for a 3D model according to an embodiment of the present invention automatically calculates and corrects an error between a virtual image and a captured image of the 3D model, the operator's post-correction work for image registration is separately The advantage is that it is not necessary.

1 is a reference diagram for explaining a photographing apparatus 10 that can be used in an apparatus for matching an image for a 3D model according to an embodiment of the present invention.
2 is a block diagram illustrating an image matching apparatus for a 3D model according to an embodiment of the present invention.
3 to 5 are reference diagrams for explaining an image matching method for a 3D model according to an embodiment of the present invention.
6 is a flowchart illustrating an image matching method for a 3D model according to an embodiment of the present invention.

The present invention will be described in detail with reference to the accompanying drawings as follows. Here, repeated descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations will be omitted. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In addition, the term "...unit" described in the specification means a unit that processes one or more functions or operations, which may be implemented as hardware or software or a combination of hardware and software.

1 is a reference diagram for explaining a photographing apparatus 10 that can be used in an apparatus for matching an image for a 3D model according to an embodiment of the present invention.

Referring to FIG. 1 , the photographing apparatus 10 captures an image of a facility to be precisely inspected, diagnosed, or managed in the image matching method for a three-dimensional model according to an embodiment of the present invention, and obtains image information. As a capable device, it may refer to a device capable of performing an equivalent function or including an image sensor that converts optical signals input through a provided lens into electronic signals. In FIG. 1 , the photographing device 10 is illustrated as an unmanned aerial vehicle as a preferred example of the present invention, and the unmanned aerial vehicle-shaped photographing apparatus 10 includes a GNSS/IMU for acquiring the position and posture information of the photographing apparatus 10 , and photographing. It may be configured to include a camera mounted on the device 10 and a gimbal for fixing the camera.

Also, the photographing apparatus 10 may store the acquired image information in a storage unit provided therein, or transmit it to an image matching apparatus for a 3D model or another server in real time through a wireless network. In addition, the operation of the photographing apparatus 10 may be controlled by various methods including wireless communication control.

2 is a block diagram illustrating the configuration of an image matching apparatus 100 for a 3D model according to an embodiment of the present invention.

Referring to FIG. 2 , the facility inspection system 100 according to an embodiment of the present invention includes a three-dimensional model generator 110 , a virtual image generator 120 , an error calculator 130 , and an image matching unit ( 140) may be included.

Each of the 3D model generation unit 110 , the virtual image generation unit 120 , the error calculation unit 130 , and the image matching unit 140 executes instructions such as computer programs or data stored in a computer-readable medium. The various steps described herein may be performed by including one or more processors, memories, and other suitable components for

For example, computer programs or instructions may be stored in a computer-readable medium internal to or external to the configuration.

The 3D model generator 110 may implement a 3D model of a facility in a virtual environment.

Here, the virtual environment may mean a virtual space in which a user can implement a 3D model of the facility. For example, the virtual environment may be arbitrarily created through the execution of specific software that may be installed in the user terminal, and may refer to a three-dimensional space visually expressed through the display device of the user terminal. Accordingly, the 3D model generation unit 110 may implement a 3D model of the facility in the virtual environment based on information on the 3D model of the facility previously formed by the user.

The virtual image generator 120 may generate a virtual image of the 3D model through virtual imaging in the virtual environment.

In an embodiment, the virtual image generator 120 may collect information on a photographing position and a photographing posture for each photographed image obtained by the photographing apparatus 10 .

For example, the virtual image generating unit 120 may directly receive and collect position and posture information when the photographing device 10 is photographed from the GNSS/IMU of the photographing device 10 , and on the other hand, the photographing device 10 may collect it. The location and posture information at the time of photographing may be received and collected from the storage device included in (10).

In an embodiment, the virtual image generator 120 derives a photographing position and a photographing posture in the virtual environment corresponding to the collected photographing position and photographing posture information, and based on the derived photographing position and photographing posture A virtual image of the 3D model may be generated.

For example, the virtual image generating unit 120 collects photographing position and photographing posture information for each of the captured images obtained by the photographing device 10 , so that the photographing position of each of the obtained captured images in the virtual environment and a photographing posture may be derived, and a virtual image similar to the photographed image captured by the photographing apparatus 10 may be generated by performing virtual photographing in a virtual environment based on the derived photographing position and photographing posture.

The error calculator 130 may calculate an error between the generated virtual image and a captured image obtained by photographing an actual facility by the photographing apparatus 10 . Here, even if the virtual image generating unit 120 generates a virtual image similar to the captured image captured by the photographing device 10 as described above, it is difficult to determine the position and posture through GNSS/IMU, etc. Errors may occur for various reasons, including the case of

In an embodiment, the error calculator 130 may match images determined to have the same photographing position and photographing posture among the virtual image and the photographed image, and may overlap the matched images to be projected on each other.

In this regard, FIGS. 3 and 4 are reference diagrams for explaining an image matching method for a 3D model according to an embodiment of the present invention.

Here, (a) of FIG. 3 shows one of the photographed images of the facility photographed by the photographing apparatus 10 according to an embodiment of the present invention, and (b) of FIG. 3 is the photographing apparatus 10 Shows a virtual image generated by applying the shooting position and posture to the image (a) captured by . In addition, FIG. 4 shows an example in which the above images (a) and (b) are superimposed to be projected onto each other.

As shown in FIG. 4 , the error calculator 130 may calculate an error for each of the matched images through a method of overlapping and comparing each of the matched images.

In an embodiment, the error calculator 130 may extract an outline of a facility included in the virtual image and the captured image. In this regard, FIG. 5 shows an example of an outline extracted through an image matching apparatus for a three-dimensional model of the present invention. Here, the error calculating unit 130 may extract an outline of a facility included in each image, and thus may be used to calculate an error between the overlapped images. For example, the error calculator 130 compares the outline of the captured image and the outline of the virtual image in the overlapped image to calculate the error, derives feature points for the outline of each image, and derives the An error may be calculated by comparing the coordinates between the feature points.

The image matching unit 140 may match the virtual image and the captured image by correcting the calculated error.

In an embodiment, the image matching unit 140 may calculate transformation coordinates in the virtual environment with respect to the acquired coordinates of the captured image.

In the image matching method for a three-dimensional model of the present invention, the obtained captured image can be compared on the same basis as the virtual image by calculating transform coordinates in the virtual environment for the obtained coordinates of the captured image. Here, the following Equation 1 is for converting the obtained coordinates of the captured image into coordinates in a virtual environment, and various variables may be used in this transformation process.

[Equation 1]

는 촬영 영상에 대한 변환 좌표, s는 축척, r은 y축 회전 각도, p는 x축 회전 각도, h는 z축 회전 각도,

는 x축 이동 거리,

는 y축 이동 거리,

및

는 각각 변환 이전의 촬영 영상 좌표임)(From here,

is the transformation coordinate for the captured image, s is the scale, r is the y-axis rotation angle, p is the x-axis rotation angle, h is the z-axis rotation angle,

is the x-axis travel distance,

is the y-axis travel distance,

and

are the coordinates of the captured image before transformation, respectively)

In an embodiment, the image matching unit 140 may match the virtual image and the captured image based on a projection transformation method using the extracted outline.

In an embodiment, the image matching unit 140 may register the virtual image and the captured image by applying the projection transformation method to each of the overlapping images.

The image registration method for a 3D model of the present invention may perform correction to minimize an error occurring between images based on a projection transformation method using the extracted outline. Specifically, for example, the image matching unit 140 derives an arbitrary feature point for the extracted outline and applies a weight matrix calculated from the slope of the outline including the feature point to the feature point, thereby positioning between the feature points. error can be minimized.

6 is a flowchart illustrating an image matching method for a 3D model according to an embodiment of the present invention.

Referring to FIG. 6 , a user may acquire a photographed image of a facility to be managed through a photographing device, and create a three-dimensional model of the facility to be managed through a specific software and display it in a virtual environment. It can be implemented (S601).

Next, the virtual image generator may generate a virtual image by photographing a facility implemented in a virtual environment to correspond to the position and posture information of the acquired image (S602).

Next, the error calculator may calculate an error between the generated virtual image and the previously acquired image (S603).

Next, the virtual image and the captured image can be matched by correcting the calculated error through the image matching unit, and through this, the user can check and manage the actual facility using the 3D model without a separate post-correction process. (S604).

The above-described image registration method for the three-dimensional model has been described with reference to the flowchart shown in the drawings. For simplicity, the method has been shown and described as a series of blocks, but the invention is not limited to the order of the blocks, and some blocks may occur with other blocks in an order different from or concurrently with those shown and described herein. Also, various other branches, flow paths, and orders of blocks may be implemented that achieve the same or similar result. In addition, not all illustrated blocks may be required for implementation of the methods described herein.

Although the above has been described with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as described in the claims below. You will understand that it can be done.

10: shooting device
100: image matching device for 3D model
110: 3D model generation unit
120: virtual image generating unit
130: error calculation unit
140: image matching unit

Claims (8)

In the image registration method for the 3D model performed in the image registration apparatus for the 3D model,
implementing a three-dimensional model of a facility in a virtual environment;
generating a virtual image of the 3D model through virtual imaging in the virtual environment;
calculating an error between the generated virtual image and a photographed image obtained by photographing an actual facility by the photographing apparatus; and
Compensating the calculated error to match the virtual image and the captured image; including,
Image registration method for 3D model.
According to claim 1,
generating a virtual image for the 3D model;
collecting, by the photographing device, photographing position and photographing posture information for each of the captured images; and
deriving a photographing position and photographing posture in the virtual environment corresponding to the collected photographing position and photographing posture information, and generating a virtual image for the 3D model based on the derived photographing position and photographing posture; characterized in that it comprises,
Image registration method for 3D model.
According to claim 1,
The step of calculating the error is
Extracting an outline of a facility included in the virtual image and the captured image; characterized in that it comprises,
Image registration method for 3D model.
4. The method of claim 3,
Matching the virtual image and the captured image;
matching the virtual image and the captured image based on a projection transformation method using the extracted outline;
Image registration method for 3D model.
5. The method of claim 4,
The step of calculating the error is
matching images determined to have the same shooting position and shooting posture among the virtual image and the captured image; and
Superimposing the matched images to be projected on each other; characterized in that it comprises,
Image registration method for 3D model.
6. The method of claim 5,
Matching the virtual image and the captured image;
and matching the virtual image and the captured image by applying the projection transformation method to each of the overlapping images.
Image registration method for 3D model.
5. The method of claim 4,
Matching the virtual image and the captured image based on the projection transformation method;
Calculating transformation coordinates in the virtual environment for the acquired coordinates of the captured image; characterized in that it comprises,
Image registration method for 3D model.
In the image registration apparatus for a three-dimensional model for performing the image registration method for the three-dimensional model,
a three-dimensional model generator that implements a three-dimensional model of a facility in a virtual environment;
a virtual image generating unit generating a virtual image of the 3D model through virtual imaging in the virtual environment;
an error calculation unit for calculating an error between the generated virtual image and a captured image obtained by photographing an actual facility by the photographing apparatus; and
An image matching unit for matching the virtual image and the captured image by correcting the calculated error;
Image matching device for 3D model.

Images