KR20210055260A - Apparatus and method for estimating camera pose and depth using images continuously acquired at various viewpoints - Google Patents

Abstract

According to the present invention, provided are a device and a method for estimating the posture and the depth of a camera. The device of the present invention comprises: a frame providing unit for selecting a key frame and a reference frame from a plurality of frames continuously acquired at various viewpoints; a depth determining unit applied with at least one of the key frame and the reference frame to acquire a depth map corresponding to the applied frame in accordance with a pre-learned pattern estimating method and pattern restoring method; and the posture determining unit for obtaining a combined feature map by combining a key feature map and a reference feature map obtained by extracting a feature in accordance with the pre-learned pattern estimating method from the key frame and the reference frame, obtaining a three-dimensional feature map by combining a three-dimensional coordinate system feature map generated based on a depth map obtained to include a three-dimensional coordinate in the combined feature map, and estimating a camera posture change from the key frame to the reference frame in accordance with the pre-learned pattern estimating method from the three-dimensional feature map. Accordingly, the posture of a camera can be accurately estimated.

Description

Apparatus and method for estimating camera pose and depth using images continuously acquired at various viewpoints}

The present invention relates to an apparatus and method for estimating camera posture and depth, and to an apparatus and method for estimating camera posture and depth using images continuously acquired at various viewpoints.

To obtain a depth image, an expensive lidar sensor is mainly used, or a corresponding pair is searched from a stereo image (left and right images) acquired at the same time, and a disparity map is obtained based on the searched corresponding pair. After that, the actual depth value is obtained through a known baseline between stereo cameras. Due to the high cost of lidar, a method of acquiring a depth image from a stereo image is generally used.

A method of obtaining distance information, that is, a depth image, from a stereo image is performed including a camera calibration step of estimating parameters of two cameras and a stereo matching step of finding positions corresponding to each other on the images acquired from the two cameras. z) is obtained according to the equation z = (b × f) / d.

Here, b is a distance between stereo cameras (baseline), f is a focal length, which is an internal parameter of the camera, and d is a disparity. Since the distance b and the focal length f between the cameras have fixed constant values, the depth value z may be calculated according to the disparity d to obtain a depth image. However, since the depth value (z) is calculated using only the disparity (d), there is a limitation in that it is not possible to obtain a range of various depth images.

In addition, there is a problem that post-processing is required because the depth value is predicted using only the left image and the right image.

In order to solve this problem, a technique for using images acquired from multiple viewpoints at the same time has been proposed, but in order to apply this technique, there is a problem that the position and posture of the camera that acquires each image must be accurately known.

Korean Patent Publication No. 10-2019-0032532 (published on Mar. 27, 2019)

An object of the present invention is to provide a camera posture and depth estimation apparatus and method capable of accurately estimating a camera posture from images continuously acquired at various viewpoints.

Another object of the present invention is to provide an apparatus and method for estimating a camera posture and depth capable of obtaining an accurate depth image by warping and comparing images continuously acquired at various viewpoints based on an estimated camera posture.

Another object of the present invention is a camera capable of acquiring camera posture and depth information in which errors are not accumulated by selecting a keyframe from images continuously acquired at various viewpoints and comparing the difference between the selected keyframe and a reference image It is to provide an apparatus and method for estimating posture and depth.

A camera posture and depth estimation apparatus according to an embodiment of the present invention for achieving the above object includes: a frame providing unit for selecting a key frame and a reference frame from images of multiple frames successively acquired at various viewpoints; A depth determination unit that receives at least one of the key frame and the reference frame and obtains a depth map corresponding to the applied frame according to a previously learned pattern estimation method and a pattern restoration method; And combining a key feature map and a reference feature map obtained by extracting features according to a pattern estimation method learned in advance from the key frame and the reference frame to obtain a combined feature map, and 3D coordinates are included in the combined feature map. A three-dimensional feature map is obtained by combining a three-dimensional coordinate system feature map generated based on the obtained depth map, and a camera from the key frame to the reference frame according to a pattern estimation method learned in advance from the three-dimensional feature map. And a posture determination unit that estimates posture change.

The posture determination unit receives the key frame and the reference frame, extracts features of each of the key frame and the reference frame according to a previously learned pattern estimation method, and obtains the key feature map and the reference feature map. Extraction unit; A feature combiner configured to obtain the combined feature map by receiving and combining the key feature map and the reference feature map; The three-dimensional feature map obtained by deforming and weighting the depth map obtained by the depth determination unit to the two-dimensional coordinate system feature map having a designated pattern according to the camera internal parameter is combined with the combined feature map to form a three-dimensional feature map. A three-dimensional conversion unit to obtain; And a posture change in which the three-dimensional feature map is applied and the rotation change and movement change of the camera between the key frame and the reference frame are separately estimated, and the camera posture change is determined from the estimated rotation change and movement change of the camera. It may include an estimation unit.

The feature extractor includes a key frame feature extractor configured to obtain the key feature map by receiving the key frame; And a reference frame feature extractor configured with the same neural network as the key frame feature extractor, trained to have the same weight, and obtain the reference feature map by receiving the reference frame.

The 3D transform unit has a first channel having a size corresponding to the combined feature map and having a predetermined pattern value in order to set a coordinate value in the x-axis direction and a predetermined pattern value in order to set a coordinate value in the y-axis direction. The 3D coordinate system feature map may be obtained by multiplying a matrix in which a second channel having a second channel and a third channel, which is a homogeneous coordinate system having a predetermined value, by the inverse matrix of a calibration matrix obtained in advance as the camera internal parameter and the depth map. .

The posture change estimator receives the 3D feature map and estimates a rotation change amount of the camera from the key frame to the reference frame according to a previously learned pattern estimation method; A motion vector extracting unit that receives the 3D feature map and estimates a parallel movement amount of the camera from the key frame to the reference frame according to a previously learned pattern estimation method; And a camera posture acquisition unit determining a camera posture change from the key frame to the reference frame based on the rotation change amount and the parallel movement amount.

The rotation transformation extraction unit may obtain a rotation change amount of the camera from the 3D feature map as a rotation matrix having a predetermined size to indicate rotation in each of the x, y, and z axis directions.

The rotation transformation extraction unit estimates the rotational variation of the camera from the three-dimensional feature map as three Euler angles (ψ, φ, θ), and calculates the estimated three Euler angles according to a known method. It can be converted to a matrix.

The frame providing unit includes an image acquisition unit that acquires images of multiple frames continuously acquired at various viewpoints; And selecting a previous frame as the key frame based on a reference frame determined to be greater than or equal to a predetermined reference value in which at least one of a previously selected key frame and a camera posture or depth difference in the multi-frame image, and a subsequent frame as the reference frame. It may include a frame selection unit to select.

The camera posture and depth estimation apparatus warps one of the key frame or the reference frame to correspond to another frame using the depth map obtained by the depth determination unit and the camera posture change obtained by the posture determination unit, and warping A warping unit for compensating the depth information of the depth map by comparing the generated frame with other frames may be further included.

The camera posture and depth estimation apparatus acquires a difference between a frame warped by the warping unit and another frame as an error at the time of learning, and a learning unit that learns by backpropagating the obtained error to the depth determination unit and the posture determination unit. It may contain more.

A method for estimating camera posture and depth according to another embodiment of the present invention for achieving the above object includes the steps of selecting a key frame and a reference frame from images of multiple frames successively acquired at various viewpoints; Receiving at least one of the key frame and the reference frame and obtaining a depth map corresponding to the applied frame according to a previously learned pattern estimation method and a pattern restoration method; And combining a key feature map and a reference feature map obtained by extracting features according to a pattern estimation method learned in advance from the key frame and the reference frame to obtain a combined feature map, and 3D coordinates are included in the combined feature map. A three-dimensional feature map is obtained by combining a three-dimensional coordinate system feature map generated based on the obtained depth map, and a camera from the key frame to the reference frame according to a pattern estimation method learned in advance from the three-dimensional feature map. And estimating the posture change.

Accordingly, the apparatus and method for estimating camera posture and depth according to an embodiment of the present invention select a key frame and a reference frame from images of multiple frames successively acquired at various viewpoints, and change the camera posture between the selected key frame and the reference frame. By acquiring a depth map of at least one of a key frame and a reference frame, and comparing and learning one of the key frame and the reference frame to correspond to the other based on the acquired camera posture change and depth map, the camera posture and the It can be trained to accurately estimate all of the depth. In particular, by combining the feature map for the key frame (key) and the reference frame (ref) with the feature map of the 3D coordinate system to estimate the camera posture change, it is possible to accurately estimate the camera posture change. In addition, by learning to estimate the Euler angle in order to estimate the rotational change among the camera posture changes, it is possible to more easily and accurately estimate the camera posture change. In addition, by selecting one of the immediately preceding or immediately following frame as a key frame based on the reference frame (ref) determined to be at least one of the previously selected key frame and the difference in camera posture or depth as a predetermined reference value or higher, It is possible to minimize the accumulation of errors.

1 shows a schematic structure of a camera posture and depth estimation apparatus according to an embodiment of the present invention.
FIG. 2 shows a difference in a camera posture determination method according to whether or not a frame selector of FIG. 1 selects a key frame.
3 is a diagram for explaining accumulation of errors due to successive frames.
4 shows a detailed structure of the camera posture determination unit of FIG. 1.
5 and 6 are views for explaining the concept of adding a 3D coordinate feature map to a 2D feature map by the 3D transformed feature extraction unit of FIG. 4.
7 shows an example of a key frame and a reference frame, a depth map for the key frame, and a reference frame.
8 shows a method of estimating a camera posture and depth according to an embodiment of the present invention.

In order to fully understand the present invention, operational advantages of the present invention, and objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a certain part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. And software.

1 shows a schematic structure of a camera posture and depth estimation apparatus according to an embodiment of the present invention, and FIG. 2 shows a difference in a camera posture determination method according to whether or not a frame selector of FIG. 1 selects a key frame, and FIG. 3 is A diagram for explaining accumulation of errors due to successive frames. 4 is a diagram illustrating a detailed structure of the camera posture determination unit of FIG. 1, and FIGS. 5 and 6 are diagrams for explaining the concept of adding a 3D coordinate feature map to a 2D feature map by the 3D transformed feature extractor of FIG. 4 7 shows an example of a key frame and a reference frame, a depth map for the key frame, and a reference frame.

Referring to FIG. 1, the camera posture and depth estimation apparatus according to the present embodiment includes a frame providing unit 100 for selecting and providing a key frame and a reference frame from continuous images of various viewpoints, and a key provided by the frame providing unit 100. The camera posture determination unit 200 for determining a change in the attitude of the camera between the frame and the reference frame, the depth determination unit 300 for obtaining depth information, and the camera posture determination unit 200 and the depth determination unit 300 A warping unit 400 for warping a reference frame into an image corresponding to a key frame based on camera posture and depth information may be included.

First, the frame providing unit 100 may include an image acquisition unit 110 and a frame selection unit 120.

The image acquisition unit 110 acquires images from different viewpoints that are continuously acquired while the camera device moves. The image acquisition unit 110 may be implemented as a camera device, and may be implemented as a storage device that stores an image acquired from a camera or a communication device that receives an image through a wired or wireless communication network.

In this embodiment, the image acquired by the image acquisition unit 110 is an image continuously acquired from a single camera, not a stereo image using a stereo camera, and in particular, may be an image acquired by changing a viewpoint according to the movement of the single camera. .

The frame selection unit 120 divides images of a plurality of frames successively acquired at various viewpoints acquired by the image acquisition unit 110 into a key frame (key) and a reference frame (ref).

The frame selector 120 selects a key frame from a multi-frame image, and selects one frame of the remaining frames except for the key frame as a reference frame ref.

The frame selector 120 first selects a key frame from a multi-frame image. The frame selection unit 120 may select the first frame or a predetermined frame among a plurality of frames as an initial key frame, and after that, at least one of a difference between a previously selected key frame and a camera posture or depth is equal to or greater than a predetermined reference value. One of the immediately preceding or immediately following frame may be selected as the key frame based on the reference frame (ref) determined by. For example, the frame selection unit 120 may select a key frame based on a warping level at which the warping unit 400 converts the reference frame ref to correspond to a key frame.

Recently, a technique for predicting a relationship between images in images such as an RGB image or an optical flow using an artificial neural network has been proposed. However, conventionally, as shown in (a) of FIG. 2, the camera posture was estimated by comparing only the difference between adjacent frames in the image of a plurality of consecutive frames. In this case, if all frames are normally photographed as shown in FIG. 3A, the camera posture can also be estimated normally without causing a large error. However, when a large change such as light spreading occurs as shown in FIG. 3(b), which is normally photographed in a specific frame among a plurality of frames, an error occurs in estimating the camera posture for the frame. In addition, since the generated error is accumulated and reflected even when estimating the camera posture in a subsequent frame, a problem of increasing continuously occurs as shown in (c) of FIG. 3. 3(c) shows the camera posture est estimated based on the difference between successive frames as shown in FIG. 2(a) compared with the actual camera posture gt, and FIG. 3(c) As shown in FIG. 1, when the camera posture is estimated based on the difference between successive frames, it can be seen that errors are accumulated and gradually increased.

Accordingly, in this embodiment, in order to reduce the accumulated error, as shown in FIG. 2(b), the frame selection unit 120 selects a key frame among a plurality of frame images, and the selected key frame ( key) to estimate the camera change in multiple frames acquired later. Therefore, errors may be accumulated in the camera posture estimated by the selected key frames, but errors occurring in at least one reference frame acquired between successively selected key frames are not transmitted when the next key frame is selected. Therefore, it is not accumulated. That is, accumulated errors can be reduced.

Here, when the frame selection unit 120 does not use a single key frame (key) from a multi-frame image, and repeatedly selects a key frame (key), the difference between the reference frame (ref) and the key frame (key) is large. This is because even if an error does not occur due to abnormal conditions such as light bleeding, the change between the key frame (key) and the reference frame (ref) increases due to the movement of the camera, making it difficult to estimate the camera posture and depth. . In addition, it is large that the frame selection unit 120 selects a frame immediately before or immediately after the reference frame ref, which is determined to be greater than or equal to a predetermined reference value in which the difference between the previously selected key frame (key) and the camera posture or depth is a key frame (key). This is to ensure that the camera posture in the subsequent frame can be stably estimated based on the frame before or after the frame in which the error has occurred. That is, this is to minimize errors that may accumulate between the selected key frames.

When the key frame (key) is selected, the frame selection unit 120 selects one of the selected key frame (key) and the frame after the selected key frame (key) as a reference frame (ref) to determine the camera attitude (200). ).

The camera posture determination unit 200 receives a key frame (key) and a reference frame (ref) from the frame selection unit 120, and estimates the changed camera posture in the reference frame (ref) based on the key frame (key). do. That is, the change in the attitude of the camera at the time when the reference frame ref is photographed is estimated based on the posture of the camera at the time when the key frame (key) is photographed.

The camera posture determination unit 200 includes a feature extracting unit 210, a feature combining unit 220, a 3D conversion unit 230, and a posture change estimation unit 240.

The feature extraction unit 210 receives the key frame (key) and the reference frame selected by the frame selection unit 120, and extracts features of each of the applied key frame (key) and reference frame according to a pre-learned pattern estimation method. Thus, a key feature map and a reference feature map are obtained.

Referring to FIG. 4, the feature extractor 210 receives a key frame feature extractor 211 for extracting a feature of a key frame and a reference frame ref to receive a reference frame. It may include a reference frame feature extraction unit 212 for extracting the feature of (ref).

Here, the key frame feature extraction unit 211 and the reference frame feature extraction unit 212 may be implemented as a siamese neural network having the same structure and the same weight. That is, the key frame feature extractor 211 and the reference frame feature extractor 212 extract features from the key frame (key) and the reference frame (ref) according to the same pattern estimation method to obtain a key feature map and a reference feature map. can do.

Each of the key frame feature extractor 211 and the reference frame feature extractor 212 may be implemented as a convolutional neural network (CNN).

The feature combining unit 220 concatenates the key feature map and the reference feature map extracted from the feature extracting unit 211 and outputs the combined feature map to the 3D transform unit 230.

The 3D transform unit 230 adds 3D coordinate information to the combined feature map applied from the feature combiner 220 and converts it into a 3D feature map.

The key feature map and the reference feature map extracted by the feature extraction unit 210 only extract and express features of the key frame (key) and the reference frame (ref), and do not provide coordinate system information. Therefore, the combined feature map in which the key feature map and the reference feature map are combined also does not provide coordinate information including depth information.

However, since the camera posture is to estimate 3D information instead of 2D, it is very difficult to estimate the camera posture change from the combined feature map as it is. Accordingly, in this embodiment, the 3D transform unit 230 adds a 3D coordinate system feature map representing 3D coordinate information to the combined feature map. That is, by adding a three-dimensional coordinate system feature map to the combined feature map, the three-dimensional world coordinate system is transformed to be included in the combined feature map. In this case, in order to include the three-dimensional world coordinate system in the combined feature map, the camera internal parameters and depth information corresponding to each pixel are required.

The camera internal parameters consist of the focal length (f _x , f _{y ) and the principal point (x 0} , y ₀ ), which is the pixel coordinate where the optical axis and image plane meet, and the camera internal parameters are obtained in advance. do. Therefore, if the depth (Z) corresponding to each pixel of the combined feature map is known, a three-dimensional world coordinate system can be extracted from the combined feature map. Accordingly, the 3D conversion unit 230 receives the combined feature map and converts it into a 3D feature map.

The 3D conversion unit 230 acquires a 3D coordinate system feature map based on the key depth information obtained for the key frame from the depth determination unit 300 or the warping unit 400, and converts the obtained 3D coordinate system feature map. Combined with the combined feature map, a 3D feature map is obtained.

Hereinafter, a method in which the 3D transform unit 230 adds a 3D coordinate system feature map will be described with reference to FIGS. 5 and 6.

In FIG. 5, (a) shows the relationship between the two-dimensional pixel coordinate system and the three-dimensional world coordinate system, and (b) shows the relationship between the camera coordinate system and the world coordinate system according to the camera internal parameter and the depth value of each pixel.

Referring to FIG. 5, the camera internal parameter is composed of a focal length (f _x , f _{y ) and a principal point (x 0} , y ₀ ), which is a pixel coordinate where an optical axis and an image plane meet, Camera internal parameters are obtained in advance. If the depth (Z) corresponding to each pixel of the 2D image is known, the 2D pixel coordinate system (x, y) is converted _{to the camera coordinate system (X c} , Y _c , Z _{c) centered on the camera according to Equation 1.} Can be converted.

And the camera internal parameter (Camera Intrinsic Parameter) is obtained in advance in the form of equation (2) as a calibration matrix (K) through calibration.

In FIG. 6, (a) shows a method in which a 3D coordinate system feature map (cof) is additionally combined with the combined feature map (fm), and (b) shows an example of a configuration of a 3D coordinate system feature map (cof). Referring to FIG. 6, a three-dimensional coordinate system feature map cof additionally coupled to the combined feature map fm may include three channels ch1 to ch3. The 2D feature map is applied in the form of a matrix having a predetermined size, and the three channels ch1 to ch3 of the 3D coordinate system feature map cof may be configured in a form corresponding to the 2D feature map. As shown in (b), the first channel (ch1) among the three channels (ch1 to ch3) is a feature map representing the x-axis coordinate system and may be composed of a matrix having a value increasing in the x-axis direction. The 2 channel ch2 is a feature map representing the y-axis coordinate system, and may be configured as a matrix having a value increasing in the y-axis direction. In addition, the third channel ch3 is a matrix representing a homogeneous coordinate system, and as an example, values of all elements may be set to 1.

However, the z-axis coordinate system cannot be expressed in three channels (ch1 to ch3). Accordingly, the 3D transform unit multiplies the depth information (D) for the key frame (key) with the inverse matrix (K ^-1 ) of the calibration matrix (K), and multiplies the three channels (ch1 ~ ch3) to obtain a 3D coordinate system feature map. Can be obtained. That is, among the three channels of the 3D coordinate system feature map cof, the first and second channels ch1 and ch2 may be viewed as a 2D coordinate system feature map for adding a 2D coordinate system to the combined feature map fm. Therefore, in this embodiment, after adding a channel corresponding to the homogeneous coordinate system feature map to the 2D coordinate system feature map, a depth map reflecting the characteristics of the camera internal parameters is added as the z-axis coordinate to obtain a 3D coordinate system feature map (cof). I can do it.

Here, the depth information on the key frame (key) may be obtained by being authorized from the depth determining unit 300 or the warping unit 400.

Depth information on the key frame (key) may be transmitted by obtaining a depth map by directly receiving the key frame (key) by the depth determination unit 300. However, as described above, in the present embodiment, the key frame is a reference value in which the difference between the previously selected key frame among the reference frames ref from which camera attitude information and depth information is obtained by the frame selection unit is determined. A frame immediately before or immediately after the above frame may be selected. That is, depth information on a key frame (key) may be obtained in advance, and depth information on a key frame (key) obtained in advance may be provided to the 3D transform unit 230.

That is, the 3D conversion unit 230 of the present embodiment adds a 3D coordinate system feature map to the combined feature map based on previously estimated depth information for a key frame and converts it into 3D, making it easier to change the camera posture. To be able to estimate.

The posture change estimating unit 240 receives the 3D feature map from the 3D transform unit 230 and estimates the posture change of the camera from the 3D feature map according to a previously learned pattern estimation method. As described above, the 3D feature map is a feature map in which a key feature map, a reference feature map, and coordinate system information for a key frame are combined. Accordingly, the rotation transformation extracting unit 241 receives the 3D feature map and estimates the change of the camera position and posture from the camera corresponding to the key frame (key) to the camera corresponding to the reference frame (ref).

The posture change estimating unit 240 may include a rotation transformation extracting unit 241, a motion vector extracting unit 242, an inverse Euler transforming unit 243, and a camera posture obtaining unit 244.

The camera posture change can be divided into a rotation change and a position change of the camera. Accordingly, the rotation conversion extraction unit 241 from the posture change estimation unit 240 receives the 3D feature map and determines the key according to the previously learned pattern estimation method. The rotational change of the camera is estimated between the frame (key) and the reference frame (ref), and the motion vector extracting unit 242 estimates the positional change of the camera.

The rotation transformation extraction unit 241 may estimate the rotation change of the camera in the form of a matrix having an Euler angle as an element. In general, rotation transformation in a 3D coordinate system is expressed as a rotation matrix having a size of 3 X 3 in consideration of rotation in each of the x, y, and z axis directions. However, as the rotation matrix having a size of 3 X 3 has 9 elements, the rotation transformation extraction unit 241 must estimate the rotation change of the camera as 9 values. However, even if an artificial neural network is used, it is difficult to construct a network that estimates nine values, and there are many constraints. Also, there is a problem that learning is not easy.

Accordingly, the rotation transformation extraction unit 241 according to the present exemplary embodiment may estimate the camera rotation change from the 3D feature map as an expression form of the Euler angle so as to easily estimate the rotation change of the camera.

Conversion from the rotation matrix R to the Euler angle may be performed according to Equation 3.

Euler angle is a proposed expression method to express the rotation change in 3D coordinates by a combination of three angles (ψ, φ, θ), so the rotation transformation extraction unit 241 learns to estimate the Euler angle from the 3D feature map. In this case, since only three angles need to be estimated, the rotation change of the camera can be easily estimated.

If the rotation transformation extraction unit 241 is learned to estimate the camera rotation change in the form of an Euler angle, the rotation transformation extraction unit 241 inversely transforms the Euler angle back into a rotation matrix (not shown). It may further include.

Each inverse Euler transform may be performed according to Equation 4.

Meanwhile, the motion vector extraction unit 242 estimates the amount of parallel movement of the camera in the x, y, and z axis directions from the 3D coordinates from the 3D feature map. Since the motion vector extractor 242 estimates the movement of the camera in the x, y, and z axis directions, it is possible to obtain a parallel motion vector having _{three elements (t x} , t _y , t _{z ).}

The camera posture acquisition unit 244 analyzes the rotation and movement changes of the camera from the rotation matrix obtained by the rotation transformation extraction unit 241 and the parallel movement vector estimated by the movement vector extraction unit 242 to generate a key frame. The camera posture change value in the compared reference frame is acquired.

Meanwhile, the depth determination unit 300 obtains a key frame depth map by receiving a key frame (key) from the frame selection unit 120 and estimating depth information of the applied key frame (key). The depth determination unit 300 may include a depth encoding unit 310 and a depth decoding unit 320.

The depth encoding unit 310 encodes the applied key frame according to a previously learned pattern estimation method. Here, the depth encoding unit 310 extracts the features of the applied key frame similar to the key frame feature extraction unit 211 and the reference frame feature extraction unit 212 of the feature extraction unit 210, but features extraction It is learned to extract features different from the unit 210. In addition, the depth decoding unit 320 receives a key frame encoded by the depth encoding unit 310, decodes a key frame encoded according to a previously learned pattern restoration method, and decodes the key frame (key). A key frame depth map indicating the depth value of each pixel is obtained.

The warping unit 400 keyframes the reference frame ref using the camera attitude change value obtained by the posture change estimation unit 240 and the depth map for the key frame (key) obtained by the depth determination unit 300. You can warp to the point of view of (key).

_{The relationship between the image coordinate system p K} of the key frame (key) and the image coordinate _{system p R} of the reference frame ref is expressed by Equation (5).

Where ~ represents the non-homogeneous coordinate system expression, T _K->R represents the warping of the key frame (key) to the image coordinate system (p _K ) and the reference frame (ref) to the image coordinate system (p _R ), and K represents the inside of the camera. Represents a calibration matrix representing a parameter, and D _K represents a depth map for a key frame.

)을 나타낸다.In FIG. 7, (a) represents a key frame image (I _K ), (b) represents a reference frame image (I _R ), and (c) represents a key frame depth map (D) estimated by the depth determination unit 300 _K ). And (d) is the warped reference frame image (

).

)과 키 프레임 영상(I_K)은 수학식 6과 같이 유사하게 표현되어야 한다.As shown in FIG. 7, an image obtained by warping a reference frame ref into a key frame by applying Equation 5 (

) And the key frame image I _K should be expressed similarly as in Equation 6.

)을 기반으로 깊이 인코딩부(310)에서 획득된 키 프레임 깊이 맵을 보정할 수 있다. 그리고 와핑부(400)는 보정된 키 프레임 깊이 맵을 참조 프레임으로 와핑하여 참조 프레임(ref)에 대한 참조 프레임 깊이 맵을 획득할 수 있다.Therefore, the warped reference image (

), the key frame depth map obtained by the depth encoding unit 310 may be corrected. In addition, the warping unit 400 may obtain a reference frame depth map for the reference frame ref by warping the corrected key frame depth map as a reference frame.

Meanwhile, the camera posture and depth estimation apparatus according to the present embodiment may further include a learning unit (not shown) for learning the camera posture determination unit 200 and the depth determination unit 300.

)과 키 프레임 영상(I_K) 사이의 차이(

(p_K) - I_K(p_K))를 손실 함수로 정의하고, 손실 함수의 값인 오차가 기지정된 기준 오차 이하가 되도록 오차를 역전파함으로써, 카메라 자세 판별부(200)와 깊이 판별부(300)를 학습시킬 수 있다.The learning part is the warping reference video (

) And the difference between the key frame image (I _{K) (}

(p _K )-I _K (p _K )) is defined as a loss function, and by backpropagating the error so that the error, which is the value of the loss function, is less than a predetermined reference error, the camera posture determination unit 200 and the depth determination unit ( 300) can be learned.

In the above, it has been described that the depth determination unit 300 estimates the key frame depth map D _K for the key frame, but the depth determination unit 300 having completed learning receives the reference frame ref, It may be configured to directly estimate the reference frame depth map D _{R from the reference frame ref.}

_{That is, the depth determination unit 300 estimates a key frame depth map (D K} ) for a key frame (key) for learning during learning, whereas when the camera posture and depth estimation device is actually used, the reference frame (ref A reference frame depth map (D _R ) for) may be estimated. _{In this case, since it is not necessary to obtain the depth map D R} for the reference frame ref from the key frame depth map D _K for the key frame, the warping unit 400 may be omitted.

In addition, even during learning, the depth determination unit 300 _{obtains a reference frame depth map D R} for the reference frame ref, and the warping unit 400 warps the key frame into the reference frame ref. You can also get an error by doing it.

8 shows a method of estimating a camera posture and depth according to an embodiment of the present invention.

Referring to FIGS. 1 to 7, the camera posture and depth estimation method of FIG. 8 will be described. First, a key frame (key) and a reference frame (ref) are selected from images of multiple frames successively acquired at various viewpoints ( S10). Here, the key frame is based on a reference frame (ref) in which the first frame or a predetermined frame is selected from among a plurality of frames of the image, or at least one of the difference between the previously selected key frame and the camera posture or depth is determined to be greater than or equal to a predetermined reference value. As a result, one of the immediately preceding or direct frame may be selected as a key frame.

Thereafter, a key feature map and a reference feature map are obtained by extracting features of each of the applied key frame (key) and the reference frame (ref) according to the previously learned pattern estimation method (S20). When the key feature map and the reference feature map are obtained, a combined feature map is obtained by combining the obtained key feature map and the reference feature map (S30).

Meanwhile, apart from the process of obtaining the combined feature map, at least one of a key frame (key) and a reference frame (ref) is authorized, and the applied frame (key) is encoded and decoded according to a previously learned pattern estimation method and a pattern restoration method. Thus, the depth map for the applied frame is estimated (S40). In the depth map estimation, for example, a key depth map for a key frame may be estimated during learning, and a reference depth map for a reference frame ref may be estimated during actual operation, but is not limited thereto.

Then, a 3D feature map is obtained by combining the 3D coordinate system feature map generated based on the estimated depth map with the obtained combined feature map and converting the combined feature map to 3D (S50).

When the 3D feature map is obtained, a camera rotation transformation amount is estimated from the acquired 3D feature map according to a previously learned pattern estimation method (S60). Here, the amount of rotation transformation may be directly estimated with a rotation matrix having a size of 3 X 3, but in the present embodiment, it is estimated as three Euler angles (ψ, φ, θ), and then through the inverse Euler angle transformation according to Equation 4 It can also be obtained as a rotation transformation matrix.

Meanwhile, apart from estimating the amount of camera rotation transformation, a camera movement vector is obtained by estimating a camera movement from a 3D feature map according to a previously learned pattern estimation method (S70).

When the rotation transformation matrix representing the amount of camera rotation transformation and the camera movement vector representing the parallel movement in the x, y, z axis directions of the camera are obtained, based on the obtained rotation transformation matrix and the camera movement vector, The camera posture change amount in the reference frame ref compared to the camera posture is determined (S80).

Meanwhile, during learning, a frame image warping step (S90) and an error calculation and backpropagation step (S100) may be further included.

In the frame image warping step (S90), the reference frame (ref) is matched to the key frame (key) based on the key depth map for the key frame (key) estimated in the depth map estimation step (S40) and the determined camera posture change amount. Warp to do. And in the error calculation and backpropagation step (S100), the difference between the warped reference frame and the key frame is calculated as an error, and the calculated error is backpropagated to the camera attitude determination unit 200 and the depth determination unit 300 to learn. Perform.

Learning may be performed repeatedly until the error becomes less than or equal to a predetermined reference error or reaches a predetermined number of times.

As a result, the camera posture and depth estimation apparatus and method according to the present embodiment selects a key frame (key) and a reference frame (ref) from images of multiple frames successively acquired at various viewpoints, and selects the selected key frame (key) and Acquire a camera posture change between reference frames (ref) and at least one of a key frame (key) and a reference frame (ref), and refer to a key frame based on the acquired camera posture change and depth map By comparing and learning one of the frames ref to correspond to the other, it is possible to learn to accurately estimate both the camera posture and the depth. In particular, by combining the feature map for the key frame (key) and the reference frame (ref) with the feature map of the 3D coordinate system to estimate the camera posture change, it is possible to accurately estimate the camera posture change. In addition, by learning to estimate the Euler angle in order to estimate the rotational change among the camera posture changes, it is possible to more easily and accurately estimate the camera posture change. In addition, by selecting one of the immediately preceding or immediately following frame as a key frame based on the reference frame (ref), in which at least one of the previously selected key frame and the difference in camera posture or depth are determined to be greater than or equal to a predetermined reference value, It is possible to minimize the accumulation of errors.

The method according to the present invention can be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: frame providing unit 110: image obtaining unit
120: frame selection unit 200: posture determination unit
210: feature extraction unit 211: key frame feature extraction unit
212: reference frame feature extraction unit 220: feature combining unit
230: 3D conversion unit 240: posture change estimation unit
241: rotation transformation extraction unit 242: motion vector extraction unit
243: camera posture acquisition unit 300: depth determination unit
310: depth encoding unit 320: depth decoding unit
400: warping part

Claims (20)

A frame providing unit that selects a key frame and a reference frame from images of multiple frames successively acquired at various viewpoints;
A depth determination unit that receives at least one of the key frame and the reference frame and obtains a depth map corresponding to the applied frame according to a previously learned pattern estimation method and a pattern restoration method; And
To obtain a combined feature map by combining a key feature map and a reference feature map obtained by extracting features according to a pattern estimation method learned in advance from the key frame and the reference frame, and to include 3D coordinates in the combined feature map A three-dimensional feature map is obtained by combining a three-dimensional coordinate system feature map generated based on the acquired depth map, and the camera posture from the key frame to the reference frame according to a pattern estimation method learned in advance from the three-dimensional feature map Camera posture and depth estimation apparatus including a posture determination unit for estimating a change. The method of claim 1, wherein the posture determination unit
A feature extractor configured to obtain the key feature map and the reference feature map by receiving the key frame and the reference frame, extracting features of the key frame and the reference frame according to a previously learned pattern estimation method;
A feature combiner configured to obtain the combined feature map by receiving and combining the key feature map and the reference feature map;
The three-dimensional feature map obtained by deforming and weighting the depth map obtained by the depth determination unit to the two-dimensional coordinate system feature map having a designated pattern according to the camera internal parameter is combined with the combined feature map to form a three-dimensional feature map. A three-dimensional conversion unit to obtain; And
Posture change estimation to determine the camera posture change from the estimated rotation change and movement change of the camera by separately estimating the rotation change and movement change of the camera between the key frame and the reference frame by receiving the 3D feature map Camera posture and depth estimation device including the unit. The method of claim 2, wherein the feature extraction unit
A key frame feature extractor configured to obtain the key feature map by receiving the key frame; And
A camera posture and depth estimation apparatus comprising a reference frame feature extractor configured with the same neural network as the key frame feature extractor, trained to have the same weight, and obtains the reference feature map by receiving the reference frame. The method of claim 2, wherein the three-dimensional transforming unit
A first channel having a size corresponding to the combined feature map and having a predetermined pattern value for setting a coordinate value in the x-axis direction and a second channel having a predetermined pattern value for setting a coordinate value in the y-axis direction, and A camera posture and depth estimation apparatus for obtaining the 3D coordinate system feature map by multiplying a matrix in which a third channel, which is a homogeneous coordinate system having a predetermined value, is combined by an inverse matrix of a calibration matrix obtained in advance as the camera internal parameter and the depth map. The method of claim 2, wherein the posture change estimation unit
A rotation transformation extraction unit that receives the 3D feature map and estimates an amount of rotation change of the camera from the key frame to the reference frame according to a previously learned pattern estimation method;
A motion vector extracting unit that receives the 3D feature map and estimates a parallel movement amount of the camera from the key frame to the reference frame according to a previously learned pattern estimation method; And
Camera posture and depth estimation apparatus including a camera posture acquisition unit determining a camera posture change from the key frame to the reference frame based on the rotation change amount and the parallel movement amount. The method of claim 5, wherein the rotation transformation extraction unit
A camera posture and depth estimation apparatus that obtains a rotational change amount of the camera from the 3D feature map as a rotation matrix having a predetermined size to indicate rotation in each of the x, y, and z axis directions. The method of claim 6, wherein the rotation transformation extraction unit
A camera that estimates the rotational variation of the camera from the 3D feature map as three Euler angles (ψ, φ, θ) and converts the estimated three Euler angles into the rotation matrix according to a known method. Posture and depth estimation device. The method of claim 1, wherein the frame providing unit
An image acquisition unit that acquires images of multiple frames successively acquired at various viewpoints; And
A previous frame is selected as the key frame, and a subsequent frame is selected as the reference frame based on a reference frame that is determined to be greater than or equal to a predetermined reference value in at least one of a previously selected key frame and a difference in camera posture or depth in the multi-frame image. Camera posture and depth estimation apparatus including a frame selection unit. The apparatus of claim 1, wherein the camera posture and depth estimation apparatus
Warping one of the key frame or the reference frame to correspond to another frame by using the depth map obtained by the depth determining unit and the camera posture change obtained by the attitude determining unit, and comparing the warped frame with another frame Camera posture and depth estimation apparatus further comprising a warping unit for correcting depth information of the depth map. The apparatus of claim 9, wherein the camera posture and depth estimation apparatus
Camera posture and depth estimation further comprising a learning unit for learning by acquiring a difference between the frame warped by the warping unit and another frame as an error, and backpropagating the obtained error to the depth determination unit and the attitude determination unit during training Device. Selecting a key frame and a reference frame from images of multiple frames successively acquired at various viewpoints;
Receiving at least one of the key frame and the reference frame and obtaining a depth map corresponding to the applied frame according to a previously learned pattern estimation method and a pattern restoration method; And
To obtain a combined feature map by combining a key feature map and a reference feature map obtained by extracting features according to a pattern estimation method learned in advance from the key frame and the reference frame, and to include 3D coordinates in the combined feature map A three-dimensional feature map is obtained by combining a three-dimensional coordinate system feature map generated based on the acquired depth map, and the camera posture from the key frame to the reference frame according to a pattern estimation method learned in advance from the three-dimensional feature map Camera posture and depth estimation method comprising the step of estimating a change. The method of claim 11, wherein estimating the posture change
Obtaining the key feature map and the reference feature map by receiving the key frame and the reference frame, extracting features of the key frame and the reference frame according to a previously learned pattern estimation method;
Obtaining the combined feature map by receiving and combining the key feature map and the reference feature map;
The three-dimensional feature map obtained by deforming and weighting the depth map obtained by the depth determination unit to the two-dimensional coordinate system feature map having a designated pattern according to the camera internal parameter is combined with the combined feature map to form a three-dimensional feature map. Obtaining; And
Receiving the 3D feature map, estimating a rotation change and a movement change of the camera between the key frame and the reference frame, respectively, and determining a camera posture change from the estimated rotation change and movement change of the camera. Camera posture and depth estimation method. The method of claim 12, wherein obtaining the reference feature map comprises:
A camera posture and depth estimation method for acquiring the key feature map corresponding to the key frame and the reference feature map corresponding to the reference frame using a Siamese neural network having the same structure and learning to have the same weight. The method of claim 12, wherein obtaining the 3D feature map comprises:
A first channel having a size corresponding to the combined feature map and having a predetermined pattern value for setting a coordinate value in the x-axis direction and a second channel having a predetermined pattern value for setting a coordinate value in the y-axis direction, and A camera posture and depth estimation method for obtaining the three-dimensional coordinate system feature map by multiplying a matrix to which a third channel, which is a homogeneous coordinate system having a predetermined value, is combined by an inverse matrix of a calibration matrix obtained in advance as the camera internal parameter and the depth map. The method of claim 12, wherein estimating the posture change
Estimating a rotation change amount of the camera from the key frame to the reference frame according to a pattern estimation method that is applied with the 3D feature map and is learned in advance;
Receiving the 3D feature map and estimating a parallel movement amount of the camera from the key frame to the reference frame according to a previously learned pattern estimation method; And
And determining a camera posture change from the key frame to the reference frame based on the rotation change amount and the parallel movement amount. The method of claim 15, wherein the step of estimating representing the amount of rotation change
A camera posture and depth estimation method for obtaining a rotational change amount of the camera from the 3D feature map as a rotation matrix having a predetermined size to indicate rotation in each of the x, y, and z axis directions. The method of claim 16, wherein the step of estimating representing the amount of rotation change
Estimating the rotational variation of the camera from the three-dimensional feature map as three Euler angles (ψ, φ, θ); And
And converting the estimated three Euler angles into the rotation matrix according to a known method. The method of claim 11, wherein selecting the frame
Acquiring images of multiple frames successively acquired at various viewpoints; And
A previous frame is selected as the key frame, and a subsequent frame is selected as the reference frame based on a reference frame that is determined to be greater than or equal to a predetermined reference value in at least one of a previously selected key frame and a difference in camera posture or depth in the multi-frame image. Camera posture and depth estimation method comprising the step of. The method of claim 11, wherein the camera posture and depth estimation method
Warping one of the key frame or the reference frame to correspond to another frame using the depth map and the camera posture change;
Comparing the warped frame with other frames to correct depth information of the depth map. The method of claim 19, wherein the camera posture and depth estimation method
The camera posture and depth estimation method further comprising acquiring a difference between the warped frame and another frame as an error, and backpropagating the acquired error to learn.

Images