CN110677556A - Image deblurring method based on camera positioning - Google Patents

Abstract

The invention discloses an image deblurring method based on camera positioning. The invention comprises two stages: a blurred image deblurring stage and a deblurred image feature extraction stage. A blurred image deblurring stage: according to the method, the depth information of the scene is obtained by utilizing the depth image of the blurred image of the current frame of the camera, and then the three-dimensional coordinates of the scene point are obtained. And obtain camera motion information, including translation and rotation, using an Inertial Measurement Unit (IMU). And calculating a fuzzy kernel of the selected block by using the data, and performing deconvolution operation by using the fuzzy kernel to obtain a deblurred image. A deblurred image feature extraction stage: this stage performs ORB feature extraction on the deblurred image and performs a subsequent SLAM process using the extracted features. The invention only carries out deblurring on the selected block, reduces the calculated amount to a certain extent and improves the operation speed.

Description

Image deblurring method based on camera positioning

Technical Field

The invention belongs to the field of computer vision, particularly relates to an image deblurring method based on camera positioning, and aims at motion blur caused by camera motion in exposure time.

Background

In the visual SLAM, a blur phenomenon occurs in a photographed image due to a shake of a camera during an exposure time. The presence of motion blur makes it difficult to perform data correlation on reconstructed landmarks and reconstruct new features, resulting in failure of the SLAM system to locate or reconstruct.

Image deblurring is an important direction in the field of image restoration, and has wide application in real life. In SLAM, more features can be provided after the image is deblurred, so that the system can better perform data association and reconstruct new features, the SLAM process can be successfully continued in subsequent frames, and the robustness of the visual SLAM on motion blur is improved.

High-quality methods for removing motion blur have been developed in recent decades, but most require a large amount of computation, and thus it is difficult to restore the image of the visual SLAM using these methods. Fergus et al obtained a good effect in the restoration of a single image based on statistical information of natural image gradients and a variational Bayesian method, but the speed was very slow due to parameter estimation, and the restoration results of some large blur kernels were not stable enough. On the basis, QiShan et al analyze the generation of the ringing effect and propose global prior and local prior, which both achieve better effects on the accuracy of fuzzy kernel estimation and the suppression of the ringing effect, but the algorithm convergence speed is not fast enough, and the setting of parameters has a large influence on the result, and the time complexity is large.

The invention mainly researches an image deblurring algorithm based on camera positioning, calculates camera motion generating motion blur by using a depth image and an Inertial Measurement Unit (IMU), and then deblurrs an image block by using the camera motion. Because the algorithm only deblurs the blocks in the image, the calculation amount is small, and the operation speed is obviously improved.

Disclosure of Invention

The invention provides an image deblurring method based on camera positioning, aiming at image blurring generated by camera shake in exposure time.

The image deblurring system is mainly divided into two stages: a blurred image deblurring stage and a deblurred image feature extraction stage.

A blurred image deblurring stage:

according to the method, the depth information of the shot scene is obtained by utilizing the depth image acquired by the depth camera of the visual SLAM system, and then the three-dimensional coordinates of the scene points are obtained. Motion information, including translation and rotation information, of the camera over the exposure time is obtained using an Inertial Measurement Unit (IMU). And calculating a fuzzy kernel of the selected image block by using the data, and performing Lucy-Richardson deconvolution operation by using the fuzzy kernel to obtain a deblurred image.

A deblurred image feature extraction stage:

in the stage, ORB feature extraction is carried out on the image processed in the blurred image deblurring stage, and the extracted features are used for carrying out the subsequent SLAM process.

The method is implemented according to the following steps:

step 1, obtaining camera motion track information which generates image blurring through an Inertial Measurement Unit (IMU), wherein the camera motion track information comprises translation and rotation information, and further calculating the motion of a camera in exposure time.

And 2, acquiring a depth image corresponding to the blurred image by using a depth camera, and performing filtering and denoising pretreatment on the depth image. And obtaining the depth information of the scene points by utilizing the processed depth image, and further constructing the three-dimensional coordinates L of the scene points.

And 3, calculating a fuzzy core of the selected image block by using the camera motion information obtained in the step 1 and the three-dimensional coordinates L of the scene points obtained in the step 2.

And 4, deblurring the selected image block by using the fuzzy kernel obtained in the step 3 through a Lucy-Richardson (LR) deconvolution algorithm.

And 5, extracting the characteristic points of the deblurred image by using an ORB characteristic extraction method, and performing a subsequent SLAM process by using the extracted characteristic points.

The method of the invention has the advantages and beneficial results that:

1. without deblurring, it is difficult to obtain sufficient localization function and the accuracy of visual SLAM is greatly reduced. The invention proposes to combine the visual SLAM with an image deblurring algorithm, and to use the information obtained in the visual SLAM for estimating a motion blur kernel by considering motion blur. By deblurring the image, the data association in visual SLAM is greatly enhanced and the localization function can be performed robustly even for blurred scenes.

2. The depth information and the camera motion information of the scene points are obtained by utilizing the depth image and the inertial measurement unit, and then the fuzzy core of the image block is calculated. From the camera motion and the three-dimensional spatial coordinates of the scene points, the motion blur kernel of the selected block can be easily predicted without any complex image processing algorithms.

3. The present invention pre-computes the blur kernel, so the deblurring problem is called non-blind deconvolution, which is simpler and faster than most blind deconvolution methods.

4. Depth information of a scene is obtained using a depth image, thereby eliminating non-uniform blur caused by camera motion and depth variation of the scene. The use of accelerometers and gyroscopes of the inertial measurement unit to improve the blurred image may help improve the accuracy and efficiency of deblurring.

5. The invention only carries out deblurring on the selected block, reduces the calculated amount to a certain extent and improves the operation speed.

Drawings

FIG. 1 is a system flow diagram of the present invention;

Detailed Description

The present invention will be described in detail with reference to specific embodiments.

The image deblurring method based on camera positioning is implemented according to the following steps.

Step 1, dividing the blurred image into small blocks of 5 × 5. Then, the Inertial Measurement Unit (IMU) is used for obtaining the camera motion information including translation and rotation information to obtain the motion P of the camera in the exposure time^k：

Where k denotes the k-th frame image, T₁Representing a transformation matrix, T, between two successive frames of images₁The data in (1) is derived from an inertial measurement unit, R is a 3 × 3 orthogonal matrix and represents rotation information, t is a three-dimensional vector, and the three dimensions represent translation in x, y and z directions respectively.

Representing a set of real numbers, SO (3) is called a special orthogonal group or a rotating matrix group.

And 2, preprocessing the depth image, including filtering and denoising. And obtaining the depth information of the scene point by using the processed depth image, namely the coordinate of the Z axis in the three-dimensional coordinate. And converting the image coordinate of a certain point in the scene into a three-dimensional coordinate L in a depth camera coordinate system by using the internal reference of the depth camera.

Wherein (u)^k,v^k) For a point on the blurred image of the k-th frame, Z^kDepth is (u) on the corresponding depth image^k,v^k) The depth value of (c). f. of_x、f_y、c_x、c_yFor the internal reference of the depth camera, the constructed matrix is called the internal reference matrix of the depth camera. (X)^k,Y^k,Z^k_ depth) is (u)^k,v^k) Corresponding to coordinate values in the depth camera coordinate system, i.e. L ═ X^k,Y^k,Z^k_depth,1)^T。

Step 3, using the camera motion P obtained in step 1^kAnd the scene point three obtained in the step 2The dimension coordinate L is calculated to obtain (u)^k-1,v^k-1). And then calculating a fuzzy kernel of a selected block in the fuzzy image, and defining the fuzzy kernel k as k (l, phi), wherein l is a fuzzy amount, phi is a fuzzy direction, and calculating by using the following formula:

wherein K₁Being an internal reference matrix, Z, of the depth camera^k-1And (h) is a mapping function of homogeneous coordinates to inhomogeneous coordinates.

And 4, deblurring the block by using the fuzzy kernel obtained in the step 3 through a Lucy-Richardson (LR) deconvolution algorithm.

And 5, extracting feature points from the deblurred block on the obtained deblurred image by using an ORB feature extraction method, and performing a subsequent SLAM process by using the extracted features.

Claims (4)

1. An image deblurring method based on camera positioning is characterized by comprising the following steps:

step 1, obtaining camera motion track information for generating image blur through an inertial measurement unit, wherein the camera motion track information comprises translation and rotation information, and further calculating the motion of a camera in exposure time;

step 2, acquiring a depth image corresponding to the blurred image by using a depth camera, and performing filtering and denoising pretreatment on the depth image; obtaining depth information of the scene points by using the processed depth images, and further constructing three-dimensional coordinates L of the scene points;

step 3, calculating a fuzzy core of the selected image block by using the camera motion information obtained in the step 1 and the three-dimensional coordinates L of the scene points obtained in the step 2;

step 4, deblurring the selected image block by using the fuzzy kernel obtained in the step 3 through a Lucy-Richardson (LR) deconvolution algorithm;

and 5, extracting the characteristic points of the deblurred image by using an ORB characteristic extraction method, and performing a subsequent SLAM process by using the extracted characteristic points.

2. The method according to claim 1, wherein the step 1 is implemented as follows:

dividing the blurred image into small blocks of 5 multiplied by 5; then, the inertial measurement unit is used for obtaining the camera motion information including translation and rotation information to obtain the motion P of the camera in the exposure time^k：

Where k denotes the k-th frame image, T₁Representing a transformation matrix, T, between two successive frames of images₁The data in the method is derived from an inertial measurement unit, R is a 3 x 3 orthogonal matrix and represents rotation information, t is a three-dimensional vector, and the three dimensions respectively represent translation in x, y and z directions;

representing a set of real numbers, SO (3) is called a special orthogonal group or a rotating matrix group.

3. The method according to claim 2, wherein the step 2 is implemented as follows:

preprocessing the depth image, including filtering and denoising; obtaining depth information of the scene points by using the processed depth image, namely a coordinate of a Z axis in a three-dimensional coordinate; converting the image coordinate of a certain point in the scene into a three-dimensional coordinate L under a depth camera coordinate system by using the internal reference of the depth camera;

wherein (u)^k,v^k) For a point on the blurred image of the k-th frame, Z^kDepth is (u) on the corresponding depth image^k,v^k) The depth value of (d); f. of_x、f_y、c_x、c_yThe constructed matrix is called an internal reference matrix of the depth camera; (X)^k,Y^k,Z^k_ depth) is (u)^k,v^k) Corresponding to coordinate values in the depth camera coordinate system, i.e. L ═ X^k,Y^k,Z^k_depth,1)^T。

4. The image deblurring method based on camera positioning as claimed in claim 3, wherein the step 3 is implemented as follows:

using the camera motion P obtained in step 1^kAnd (u) is obtained by calculating the three-dimensional coordinate L of the scene point obtained in the step 2^k-1,v^k-1) (ii) a And then calculating a fuzzy kernel of a selected block in the fuzzy image, and defining the fuzzy kernel k as k (l, phi), wherein l is a fuzzy amount, phi is a fuzzy direction, and calculating by using the following formula:

wherein K₁Being an internal reference matrix, Z, of the depth camera^k-1And (h) is a mapping function of homogeneous coordinates to inhomogeneous coordinates.

Images