WO2014183385A1

WO2014183385A1 - Terminal and image processing method therefor

Info

Publication number: WO2014183385A1
Application number: PCT/CN2013/085782
Authority: WO
Inventors: 刘冬梅; 刘凤鹏
Original assignee: 中兴通讯股份有限公司
Priority date: 2013-05-17
Filing date: 2013-10-23
Publication date: 2014-11-20
Also published as: CN104166972A

Abstract

A terminal and an image processing method therefor. The method comprises: an image obtaining step: obtaining two images with an overlapped region; and an image combination step: registering the two images according to the overlapped region, and synthesizing the registered two images. In the method of the present invention, two groups of images which are shot in different angles and have an overlapped region, a feature point parameter is directly extracted for each image, and a matching degree between images is determined according to each feature point, an error matching pair is removed, and combination integration processing is performed according to the registered image.

Description

Terminal and method for realizing image processing thereof

Technical field

The present invention relates to the field of image processing technologies, and in particular, to a terminal and a method for implementing image processing.

Background technique

With the rapid development of mobile device technology, the functions of mobile devices are becoming more and more powerful. It is an indispensable part of people's lives. With the powerful functions, people's requirements for hearing and visual enjoyment are getting higher and higher, but at present, mobile device terminal cameras can only capture a single picture. When using a normal mobile device camera to acquire a scene image of a field of view, Even on high-end mobile devices, people have to adjust the focal length of the camera. By zooming the lens, the full scene can be taken, but the resolution of the captured image is relatively low, because the resolution of the camera is certain, and the larger the scene is. The lower the resolution (that is, the same resolution, the picture will be blurred when the scene is large, and the scene will be clear when the scene is small); in order to obtain high-resolution scene photos, the camera lens has to be zoomed to reduce the field of view. , but this does not get a complete picture of the scene, so you need to compromise between the size and resolution of the scene. It can be seen that the current terminal has certain functional defects in the way of ingesting photos, which cannot meet the user's use requirements. Summary of the invention

The embodiment of the present invention provides a terminal and a method for realizing image processing thereof, so that an image taken by the terminal can meet the dual requirements of visual field and resolution.

According to an aspect of the embodiments of the present invention, a method for implementing image processing by a terminal is provided, including:

Image acquisition step: acquiring two images with overlapping regions;

Image matching step: According to the overlapping area, the two images are registered, and the two images after registration are combined. Optionally, in the method of the embodiment of the present invention, the registering the two images according to the overlapping area includes: extracting feature points of the two images, and extracting two images from the feature points. Matching feature pairs, with the matching feature pairs as alignment points, register the two images.

Optionally, in the method of the embodiment of the present invention, the feature point includes a corner point of the image.

Optionally, in the method of the embodiment of the present invention, the extracting feature points of the two images includes: for each image, convolving with the image by using a 3 x 3 convolution kernel to obtain each pixel of the image a partial derivative, and using the partial derivative to calculate a symmetric matrix M in a Plessy corner detection algorithm corresponding to each pixel point;

Set the selection window, and the feature point available evaluation function R; where R = ^Det(M) ,

Trace(M) + ε Det(M, = λ, . Trace(M) = + λ, , , are the eigenvalues of the matrix Μ, respectively, ε is the minimum value that makes the denominator non-zero; Selecting a detection area on the image, selecting a pixel with the largest R value in the detection area, and moving the selection window until the complete image is filtered;

The feature point determination threshold is set, and the pixel points whose R values are larger than the determination threshold are selected as the extracted feature points.

Optionally, the method of the embodiment of the present invention further includes:

Before the feature point is extracted, the boundary feature points in the image are deleted by using a preset boundary template;

And/or, after extracting feature points in the image, extracting sub-pixel feature points in each feature point, and extracting the sub-pixel feature points as the final extracted feature points.

Optionally, in the method of the embodiment of the present invention, the matching feature pairs of the two images are extracted from the feature points, including:

Using the coarse matching bidirectional maximum correlation coefficient BGCC algorithm, the feature points in the two images are coarsely matched, and the matching feature pairs obtained by the rough matching are accurately matched by the random sample RANSAC algorithm to obtain the accurately extracted matching feature pairs.

Optionally, in the method of the embodiment of the present invention, the feature points in the two images are coarsely Before the matching, the method further includes:

The two images are smoothed by the median filter, and the result of subtracting the original image from the filtered image is used as the operation object of the rough matching processing.

Optionally, in the method of the embodiment of the present invention, the synthesizing the two images after registration comprises: grading gray of each pixel of the two images after registration according to the progressive progressive synthesis method value

/(X, set; where, the setting rules include:

f(x,y) = , (x, y) e ( nf ₂ )

, d _x < d ₂ , (x, y) e (f, nf ₂ )

Where , Λ (representing the gray value of the pixel in the two images, 4, ^ (ο, ι), and 4 +^ = 1 respectively, indicating the progressive factor of the two images, ifo ₀ r is preset The determination threshold value, ; / ₂ respectively represent two images. Optionally, in the method of the embodiment of the invention, the synthesizing the two images after the registration further comprises:

The 7 X 7 area around the seam is the seam processing area, and the pixels in the seam processing area are linearly filtered by the 3 x 3 template.

Optionally, in the method of the embodiment of the present invention, between the image obtaining step and the image combining step, the method further includes:

Image pre-processing step: processing the two images acquired by the image obtaining step according to a set pre-processing operation; wherein, the pre-processing operation comprises one or more of the following operations: verifying the acquired image, and The image is converted into the same coordinate system, the two images are smoothed and processed, and the initial positioning is performed to obtain a rough overlapping region, and the overlapping region is taken as an extraction region of the feature point.

a 3D image generating step: acquiring a composite image and another image having an overlapping area with the composite image, performing the image combining step to perform re-synthesis, repeating the image capturing and image combining process, and obtaining a 3D image having a depth of field; wherein, Composite image has another image with overlapping regions It can be a non-synthetic image or a composite image.

According to another aspect of the embodiments of the present invention, a terminal is provided, including:

An image acquisition module configured to acquire two images having overlapping regions;

The image matching module is arranged to register the two images according to the overlapping area, and synthesize the two images after registration.

Optionally, in the terminal according to the embodiment of the present invention, the image matching module is configured to extract feature points of two images, and extract matching feature pairs of two images in the feature points, to match the The feature pair is the alignment point, and the two images are registered.

Optionally, in the terminal according to the embodiment of the present invention, in the image matching module, the feature point includes a corner point of the image.

Optionally, in the terminal according to the embodiment of the present invention, the image matching module further includes: a calculation submodule configured to convolve with the image by using a 3 x 3 convolution kernel for each image, and obtain an image The partial derivative of the pixel, and using the partial derivative to calculate the symmetric matrix M in the Plessy corner detection algorithm corresponding to each pixel point;

Setting a sub-module, which is set to set a selection window, and a feature point available evaluation function R;

Det(M)

In , R , where /^ (Λ^^ Ι^, Trace(M) = + l ₁ , , respectively

The characteristic value of M, £· is a minimum value that makes the denominator non-zero; the screening sub-module is arranged to select a detection area on the image according to the selection window, and the R value is selected to be the largest in the detection area. Pixels, move the selection window until the full image is filtered;

The extraction sub-module is configured to set a feature point determination threshold, and set, in the selected pixel points, a pixel point whose R value is greater than the determination threshold value as the extracted feature point.

Optionally, in the terminal according to the embodiment of the present invention, the image matching module further includes: a coarse matching submodule configured to perform rough matching on feature points in the two images by using a coarse matching bidirectional maximum correlation coefficient BGCC algorithm ; as well as

Accurately matching sub-modules, which are set to use a random sample RANSAC algorithm for coarse matching The matching feature pairs are accurately matched to obtain an accurately extracted matching feature pair.

Optionally, in the terminal according to the embodiment of the present invention, the image matching module is further configured to set a gray value/(X, a pixel value of each pixel of the two images after registration according to a progressive fade-out synthesis method. ; The setting rules include:

f(x, y) =

Where , Λ (representing the gray value of the pixel in the two images, 4, ^ (ο, ι), and 4 + ^ = 1 respectively, indicating the progressive factor of the two images, ifo ₀ r is preset The determination thresholds, and / ₂ respectively represent two images. Optionally, the terminal according to the embodiment of the present invention further includes: an image preprocessing module, and/or a 3D image generation module, where:

The image pre-processing module is configured to process the two images acquired by the image acquisition module according to a set pre-processing operation; wherein the pre-processing operation comprises one or more of the following operations: Image, converting the two images into the same coordinate system, smoothing and filtering the two images, and initially positioning, obtaining a substantially overlapping region, and extracting regions with the overlapping regions as feature points;

The 3D image generation module is configured to acquire a composite image and another image having an overlapping area with the composite image, trigger the image matching module to perform re-synthesis, repeat image acquisition and image matching process, and obtain a depth of field The 3D image; wherein, another image having an overlapping area with the composite image may be a non-composite image or a composite image.

The beneficial effects of the embodiments of the present invention are as follows:

The terminal and the method according to the embodiment of the present invention extract feature point parameters directly for each image by taking two sets of images with different angles but overlapping areas, and then determine the matching degree between the images according to each feature point, and eliminate the wrong matching pair. The composite image is processed according to the image after registration, and a wide field of view and high resolution image are obtained, which greatly improves the user experience; According to the terminal and method of the embodiment of the present invention, the series of images are spatially overlapped by taking a series of images with different angles but overlapping regions to form a wide viewing angle scene including each image sequence information. High-definition new images with 3D effects, which not only meet the requirements of wide field of view and high resolution, but also better meet the needs of users. BRIEF abstract

The drawings used in the embodiments or the related art description will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art do not pay Other drawings can also be obtained from these drawings on the premise of creative labor.

FIG. 1 is a flowchart of a method for implementing image processing by a terminal according to Embodiment 1 of the present invention; FIG. 2 is a schematic diagram of geometric matching points of an actual matching point and an estimated point according to Embodiment 2 of the present invention; The overall processing framework of image processing in the second;

4 is a flowchart of a method for implementing image processing in a terminal according to Embodiment 3 of the present invention; FIG. 5 is a processing framework diagram of an application example according to Embodiment 3 of the present invention;

6 is a structural block diagram of a terminal according to Embodiment 4 of the present invention;

FIG. 7 is a structural block diagram of a terminal according to Embodiment 5 of the present invention.

Preferred embodiment of the invention

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The present invention provides a terminal and a method for realizing image processing thereof, and a method for realizing image processing by using a single picture captured by a current terminal device. The image is spatially overlapped and synthesized, and a scene image with a large field of view is obtained without reducing the image resolution. The specific implementation process of the present invention will be described in detail below through several specific embodiments. Embodiment 1

An embodiment of the present invention provides a method for implementing image processing in a terminal. As shown in FIG. 1 , the method includes: Step S101: acquiring two images having overlapping regions;

In this step, the acquired image may be an image stored in a terminal storage module (such as a terminal internal memory and/or an external expansion memory), or may be an image captured by the terminal in real time, and the embodiment of the present invention does not perform image acquisition. The only limit.

When the terminal acquires an image by means of real-time ingestion, the embodiment provides a preferred implementation manner, which is specifically: setting two sets of rotatable cameras in the terminal, since the two sets of cameras can change the angle to shoot Therefore, images with different angles and overlapping regions can be acquired at the same time. This acquisition method provides strong support for speeding up image processing.

Step S102, registering the two images according to the overlapping area.

In this step, the two images are registered according to the overlapping area, including: extracting feature points of the two images, and extracting matching feature pairs of the two images in the feature points, wherein the matching feature pairs are Align the points and register the two images.

Wherein, the feature points may be any geometric or grayscale features suitable for image matching extracted according to image properties. The embodiment of the present invention preferably uses a corner point as a feature point to be extracted.

For the extraction of corner points, the corner point detection algorithm is mainly used. The corner point detection algorithm is mainly divided into two types of edge-based and gray-based extraction algorithms. Because the former has a large dependence on edge extraction, if it is detected, An error occurs at the edge of the edge or the edge line is interrupted (this is often the case in practice), which will have a greater impact on the corner extraction result, while the grayscale-based extraction algorithm mainly calculates the grayscale in the local range. And the violent maximum point of the gradient changes to achieve the purpose of detection, without edge extraction, and thus has been widely used in practice. The most representative corner detection algorithms are: Moravec operator corner detection, Forstner operator corner detection, Susan detection algorithm, Plessy corner detection algorithm. The Plessy corner detection algorithm has excellent performance in terms of consistency and effectiveness, and the extracted corner points are proved to have the advantages of rotation, translation invariance, and stability.

Among them, the basic idea of the Plessy corner detection algorithm is to determine the corner point by using the gray rate of change of the image. The method calculates the eigenvalue of the matrix M associated with the autocorrelation function of the image, ie The first-order curvature of the autocorrelation function determines whether the point is a corner point. If both curvature values are high, then the point is considered to be a corner point.

The Pless corner detection algorithm defines the autocorrelation value ^£ ( ^M , ^V ) in any direction as:

I _x and I _y are gradient values in the x and y directions of the image, respectively, and σ is a parameter representing the width of the Gaussian filter, which represents a convolution operation. Μ is a 2 x 2 symmetric matrix, so there must be two eigenvalues 4 and eigenvalues reflecting the characteristics of the image pixels, ie if the pixel points ( ^χ ' is a feature point, then 2 Μ matrices of this point The eigenvalues are all positive values, and they are local maxima in the region centered on ( ^χ ', then the feature points can be expressed as evaluation functions:

R = Det(M) - kTrace ² (M) (2) where DM, = , Trace{M) = l ₁ + l _{2 ?} I ^ is the determinant of the matrix, Trace is the trace of the matrix (matrix diagonal) The sum of the elements). Set a reasonable threshold T. When the R calculated by the formula (2) is greater than the threshold, it means that a corner is found, otherwise it is not. The feature points are generally pixel points corresponding to the maximum interest values in the local range. Therefore, after calculating the R value of each point, non-maximum value suppression is performed, and all points with the largest local interest value in the original image are extracted. Where k is an experimental value, generally k = 0.04 ~ 0.06.

In step S102, extracting matching feature pairs of the two images includes:

There are quite a few redundant points in the feature points extracted from the two images. If these redundant feature points are not removed, the errors of the matching parameters will be caused, and even the matching will fail. Choosing the appropriate point matching criterion to find the corresponding feature pairs is an important guarantee for the correctness and accuracy of image matching. Common methods for matching feature pairs include, but are not limited to: Hausdorff distance method, relaxation mark method, deterministic annealing algorithm and iteration Nearest Point Algorithm (ICP).

Step S103, synthesizing the two images after registration.

Specifically, after spatially registering the two images, it is necessary to select an appropriate map. Like a compositing strategy, complete the blending of images. The so-called image synthesis is to combine the pixels of the source image to generate pixels on the matching plane, and realize the natural transition between adjacent images. Among them, the selected synthesis strategy should be able to minimize the influence of the residual deformation and the difference of brightness between images on the combined effect, in order to obtain a more accurate, more comprehensive and more reliable image description of the same scene. Based on the above selection criteria, the image synthesis strategy selected in the embodiment of the present invention may be, but is not limited to, a progressive fade synthesis method.

In summary, the method in this embodiment extracts feature point parameters directly for each image by taking two sets of images with different angles but overlapping areas, and then determines the matching degree between the images according to each feature point, and eliminates the wrong matching pair. According to the image after registration, the composite image is processed to obtain a wide-field, high-resolution image, which greatly improves the user experience. Embodiment 2 The embodiment of the present invention provides a method for implementing image processing in a terminal. The method in this embodiment provides several improvements under the main structure described in Embodiment 1, which can further speed up image processing speed and accuracy. , continue as shown in Figure 1, including the following steps:

Step S101, acquiring two images having overlapping regions;

The implementation process of this step is the same as that of the first embodiment, and detailed description thereof will not be repeated.

Step S102, registering the two images according to the overlapping area.

In this step, with regard to the extraction of feature points, an improved Plessy corner detection algorithm is used, specifically:

In view of some defects such as the original Plessy corner detection single threshold setting, low positioning accuracy and poor real-time performance, this embodiment proposes several improvements to extract as many feature points as possible and accurate in the image, and accelerate at the same time Extract the speed of the corner points. In conjunction with the implementation of the Plessy corner detection algorithm described in the first embodiment, the implementation process of the improved Plessy corner detection algorithm includes:

1) Calculate the first-order partial derivatives I _X , Iy in the horizontal and vertical directions and the product I _x I _{y of the} two points for each point of the image. Using the obtained partial derivative information, calculate the symmetric matrix according to formula (1). M.

It is not easy to find the partial derivative in image processing, and a preferred calculation method is given in this embodiment: The convolution of the 3 χ 3 convolution kernel with the original image can be used to obtain the first-order partial derivative I _x , I of each point of the original image. The 3 x 3 convolution kernel can be, but is not limited to, the following template: α

Then the first-order partial derivative 4 = - i _v = i®o is obtained.

Dx 3y 2) In the original Plessy corner detection algorithm, the k value of the feature point available in the evaluation function R is an empirical constant, which is arbitrarily used, resulting in a decrease in the reliability of corner extraction, in the case of different picture conditions. Underneath, it is easy to affect the accuracy of corner extraction. Considering that R is essentially a corner detection signal, the value of the determinant is large, and the value of the trace is small as a corner signal, and vice versa. Therefore, in the improved algorithm, the following evaluation method is used to calculate the feature point available evaluation function:

R = ^{Det M} ( 3 )

Trace(M) + ε where, in order to avoid that the matrix trace may sometimes be zero, adding a small number in the denominator avoids the selection of the parameter k compared to the evaluation function proposed in the original Plessy corner detection algorithm. The randomness of k selection has practicality, good reliability and high accuracy. Where ε takes any small number greater than zero, ε has an arbitrary degree. Since the expression is arbitrarily close, ε can take a positive value arbitrarily, but it can take any value to accurately express the meaning of "infinite proximity" in the limit definition. However, in order to highlight "infinite proximity", 0 < ε < 1 is usually taken.

3) Select local extremum points. It is customary to choose an appropriate threshold, and then use pixels with interest values greater than the threshold as feature points. Those pixels with interest values less than the threshold are filtered out. . Although this method is simple and easy to implement, the selection of a single threshold may cause some feature points to be filtered out for non-homogeneous images. In order to overcome this defect, the improved Plessy corner detection algorithm uses the in-window suppression non-maximum method in the image to combine the threshold settings to filter the feature points. The principle is: select an appropriate window in the image. The pixel with the largest R in the window is retained, and the remaining pixels in the window are deleted. The moving window filters the pixels of the entire image. The number of local extremum points is often many. A reasonable threshold is set according to requirements, and the largest number of selected pixels are selected as the final feature point extraction results. Preferably, in order to speed up the extraction, 预先 Use a preset boundary template to exclude boundary corner points that do not match well.

The above selection process is expressed in steps:

3.1) setting a selection window, selecting a detection area on the image according to the selection window, screening a pixel with the largest R value in the detection area, and moving the selection window until the complete image is filtered;

3.2) Set the feature point determination threshold, and set the pixel points of the selected pixel points whose R value is larger than the determination threshold as the extracted feature points.

The size of the "selection window" and the "judgment threshold" can be flexibly set according to actual requirements, and the main performance is that when the selection window is small, more pixels are selected; otherwise, the selected pixels are selected. less. For the judgment of the threshold, the larger the setting, the fewer the feature points are finally extracted; on the contrary, the extracted feature points are more. In the embodiment of the present invention, the decision width is 2200, and the non-maximum window is 7*7. However, the selection window and the decision threshold can be flexibly set according to requirements. This embodiment does not limit the size.

Preferably, after the feature points are extracted, a sub-pixel feature point (corner point) positioning process may also be performed, and the feature points may be further accurately extracted by positioning the sub-pixel feature point positioning process. The positioning implementation is as follows: 二次 Use the quadratic polynomial ^ + ² + ^ + + + / = , y to approximate the feature point and use the evaluation function R to achieve the sub-pixel-level precise position of the corner. Specifically, using the ordinary pixel points around the corners that have been detected, six overdetermined equations with a ~ f can be established, and the overdetermined equations are solved by the least squares method. Subpixel-level corner points correspond. Is the maximum point of the quadratic polynomial, and the pixel corresponding to the maximum point is the extracted precise feature point. In other words, when the pixel corresponding to the maximum point is the corresponding corner point in the calculation, The extracted precise feature point is the corner point; otherwise, the corresponding corner point in the calculation is deleted, and the pixel point corresponding to the maximum value point is the accurately extracted corner point (feature point).

In step S102, the following extraction methods are preferably used for matching feature pairs:

The matching algorithm proposed in this embodiment is divided into two steps: rough matching is performed by using the bidirectional maximum correlation coefficient (BGCC); then it is purified by random sampling method (RANSAC) to achieve the perfect matching of images. The method can accurately extract the correct matching feature point pairs while removing the redundant feature points.

The rough matching uses the Bidirectional Greatest Correlative Coefficient (BGCC) method to establish a similarity measure NCC. The matching is considered successful only when the two corner points are the largest relative measure value of the other party. Specifically: The correlation coefficients are defined as follows:

(2n + l)(2n + ( ₄ )

⁷ i, is the grayscale of the two images; _n x _n is the size of the window selected in one of the images; A x / is the size of the search area selected in the other image, and the angle in the first image is set The point is (3⁄4, i=l...m, the corner point in the second image is (3⁄4, j=l...n, then the ith and jth to be matched in the two figures respectively) The feature point ( ^M , ^V ) is the average gray value of the corner window area:

Standard variance of the window area

The coarse matching of the corner points by the bidirectional maximum correlation coefficient algorithm is as follows:

1) Select an nxn related window centering on any corner of the image ι, and select a rectangular search area of size X from the pixel with the same coordinates as the corner point in ι, and then ι Calculate the correlation coefficient between the corner points in the middle and the corner points in the search window area

^C uses the corner point with the largest correlation coefficient as the matching point of the given corner point, so that a set of matching points can be obtained.

2) Similarly, given any corner point in image 2, the corner point of the corresponding window region in the image ι with the largest correlation coefficient is used as the matching point of the given corner point, so that a set of Pisi can also be obtained. I have a collection.

3) Finally, the same pair of matching corner points are searched in the obtained two sets of matching points, and it is considered that the pair of corner points are matched with each other, and thus the initial matching of the corner points is completed.

In actual operation, in order to compensate for the difference between the two images due to illumination, the image is filtered with median. The waver (such as the median filter of 7 X 7) is smoothed, and then the result of subtracting the original image from the filtered image is taken as the object of the operation.

However, if only BGCC is used for matching, an incorrect matching pair will be generated. Sometimes the proportion of error matching will be very high, which seriously interferes with the estimation of the transformation matrix, resulting in image failure. Therefore, the feature point pairs must be corrected to remove the wrong matching pair. In this embodiment, the random matching method (RANSAC) is used for fine matching.

The basic idea of RANSAC is: Firstly, some kind of objective function is designed according to the specific problem, then the initial value of the parameter in the function is estimated by repeatedly extracting the minimum point set, and all the data are divided into so-called "inner points" by using these initial parameter values. "(inliers, the point that satisfies the estimated parameters) and "outliers" (points that do not satisfy the estimated parameters), and finally recalculate and estimate the parameters of the function with all "inside points". Specifically, the so-called minimum point set is sampled in the input data, and the parameter to be determined is estimated by using the minimum point set obtained by each sampling, and at the same time, according to certain discriminant criteria, which of the input data is associated with the group The parameters are consistent, that is, "inside point", which are inconsistent, that is, "out of point". After iterating a certain number of times, the estimated parameter value corresponding to the highest "inside point" ratio in the input data is taken as the final parameter estimation value.

The specific implementation process of the RANSAC algorithm applied to this embodiment is as follows:

(1) randomly select n pairs of matching points (the selected n points should ensure that any three points in the sample are not on the same line), and linearly calculate the projection transformation matrix H; where n is greater than or equal to 4

(2) Calculate the distance from each matching point to the corresponding matching point after transformation by the projection transformation matrix H; (3) Calculate the interior point according to the principle that the inner point distance is smaller than the set distance threshold value t, and select one containing the innermost point. Point set, re-estimating the projection transformation matrix H on the inner point domain;

(4) randomly select n pairs of matching points, return to step (2), and repeat N times, to obtain a more accurate projection transformation matrix H, and perform projection transformation on each matching point obtained by rough matching according to the matrix H. The inner point is the pair of matching features that are accurately extracted.

The estimated transformation matrix H requires at least eight equations, that is, n ( > 4 ) pairs of feature pairs need to be selected in the adjacent two images, and the feature pairs can be obtained through the above-mentioned corner matching process. The projection of ι, ² is transformed into (in homogeneous coordinates):

The cross product equation can be expressed as: ^X ; ^XHX i ^{= 0} where ⁼ («' ) ^: Let j line representing H, then the cross product equation can be expressed as Ah=0

0 ^T -w'X y'X

W; X 0 ^Γ -x _t 'X\ =0

-y'X ^X 0 ^T

(8) In practice, by SVD decomposition of A, the solution of h is the value of V, and then the matrix can be obtained.

H ₀

The calculation of the inner point according to the principle that the inner point distance is less than the set distance threshold value t includes: as shown in FIG. 2, respectively, *, respectively, points ^, corresponding points estimated in the respective corresponding images, then in the image The geometric distance between the actual matching point of a point and its estimated matching point is defined as follows:

d(p,p) = d(p,H~ ^l q) = \\p -H~ ^l q | , d'(q, q) = d(q, Hp) = \\q-Hp | ( g In the formula, ΙΙ·Ι indicates the Euclidean distance. Considering symmetry, the geometric distance decision criterion function is defined as follows: dis― d _{ (p _f , p') ² + d _i ^/ (q _i , q;) ² = p _f - H~q _i + 1|3⁄4 - Hp _i || , = 1, 2,..., ( 10) If the calculated dis is greater than the set distance threshold, the corresponding matching point is considered to be the outlier; if the calculated dis is less than the set distance For the value, the corresponding matching point is considered to be an inner point, and only the inner point is suitable for calculating the transformation matrix H.

Step S103, synthesizing the two images after registration.

In this step, in order to smooth the kneading region and ensure the image quality, the improved progressive progressive synthesis method is used for image synthesis, as follows:

The original progressive gradual synthesis method takes the gradation value f(x, y) of the pixel in the overlapping region of the image from the gradation values f (x, y) and f ₂ (x, y) of the corresponding pixel in the two images. The weighted average is obtained:

f(x,y) =dl xfl(x,y)+d2 χβ (x,y) where d ₂ is a gradation factor whose value is limited to between (0, 1) and satisfies 4+4=1 The relationship, in the overlapping area, from 1 to 2 in the direction from the 1st image to the 2nd image, d ₂ is ramped from 0 to 1, and f (x, y) slowly transitions smoothly to f ₂ ( x, y). However, when using this algorithm, it is found that the processed image still eliminates the boundary in the image, but still overlaps the image. The phenomenon of ghosting and blurring occurs. Because of the large difference in the gray values of the corresponding pixels in the overlapping portions of the two images, the gray value of the synthesized image appears to jump at these pixels, in order to avoid such a situation. The ioor, for f(x, y), does not directly take the weighted average of f (x, y) and f ₂ (x, y), but first calculates the pixel corresponding to the pixel in the first two graphs. The difference of the gray value, if the difference is less than the threshold, the weighted average is taken as the gray value of the point, and vice versa, the gray value before the smoothing is the gray value of the point.

The image pixel f(x, y) synthesized by the correction algorithm can be expressed as:

f(x,y) =

(11)

Where ^χ ' and 2 respectively represent the gray values of the pixels in the two images, 4, 0, 1), and 4+^=1, respectively representing the progressive factors of the two images, and f respectively representing the two images. The door is a preset judgment threshold. It can be known from the formula (11) that the judgment threshold is used to determine which gray scale definition method is used for the pixel of the overlap region. When the ioor is set too large, it will cause all Pixels | ^_/2 I may be less than the ioor, resulting in the final grayscale setting is not accurate; when the ioor is set too small, it will cause all the pixel points l^-^l value is greater than the door, it will also lead to The final grayscale setting is not allowed. Therefore, when setting the door, it is recommended to compare the gray values of some overlapping areas in advance, and find an empirical value of the gray difference value, and use the empirical value as a reference reference to adjust the ioor. Therefore, the embodiment of the present invention only proposes the concept of ioor, and the specific value of ioor is not limited.

Furthermore, in the image synthesis, if the overlap area of the selected seam is too large, problems such as image blurring and inconspicuous edge information may occur. If the overlap area of the selected seam is too small, the seaming phenomenon of the image cannot be eliminated. Therefore, in the embodiment, for the processed image, the 7x7 region around the patchwork is used as the seam processing area, and the 3x3 template is used to linearly filter the pixels in the seam region, and the effect is best.

The overall processing flow chart of this embodiment is shown in FIG. In summary, the embodiment is based on the implementation architecture of the first embodiment, and the method for extracting feature points, matching feature extraction, and image synthesis Improvements to further speed up image processing and accuracy. Embodiment 3 This embodiment provides a method for implementing image processing in a terminal. As shown in FIG. 4, the method includes the following steps:

Step S401: Acquire two images with overlapping regions.

Step S402: Perform preprocessing on the acquired two images to ensure the accuracy of the image blending in the next step. The preprocessing process includes one or more of the following processing methods:

In the first mode, it is verified whether the acquired two images have overlapping regions, and when there is an overlapping region, proceed to the next step; otherwise, an error message is sent;

In the second mode, the two images are converted into the same coordinate system to facilitate subsequent image blending processing; in the third method, the image is smoothed and filtered to provide precision support for subsequent image blending processing; and the fourth method is initially positioned to obtain approximate An overlapping area, and an extraction area characterized by the overlapping area. This preprocessing method narrows the matching range and improves the image processing speed.

Of course, the above pre-processing methods are merely exhaustive, and any operation that can be thought of by those skilled in the art to support subsequent image matching processing is within the protection concept of the present invention.

Step S403: Extract feature points of the two images, and extract matching feature pairs of the two images in the feature points, and register the two images by using the matched feature pairs as alignment points.

The implementation of this step can be implemented in the manner described in Embodiment 1 or Embodiment 2.

Step S404, synthesizing the two images after registration.

Step S405: Acquire a composite image and another image having an overlapping area with the composite image, perform steps S403 and S404 to perform re-synthesis, repeat image acquisition and image matching process, and obtain a 3D image with depth of field. The other image having a certain overlapping area with the composite image may be a non-composite image or a composite image.

A specific application implementation process using the method described in this embodiment is shown below. The basic processing framework is shown in FIG. 5, and the specific implementation process includes:

The user can select to enable the image processing function through the terminal interface;

Initialize two sets of cameras, adjust the angle of the two sets of cameras; In the image area where a certain overlapping area is ensured, two sets of pictures of different angles are obtained; image preprocessing is performed on the two sets of pictures;

The processed two sets of pictures enter the picture blending process to generate a set of composite pictures A;

The two sets of cameras are used to continue photographing to synthesize B, C, D ... and store the synthesized pictures locally;

(7) until the user has completely photographed the desired picture, and then further combines the images of the series synthesized in the storage module until a picture with a 3D effect with different depth of field is generated;

(8) The user can directly preview the generated image.

In summary, in the method of the embodiment of the present invention, the series of images are spatially overlapped by taking a series of images with different angles but overlapping regions to form a wide viewing angle scene including each image sequence information. Complete, high-definition new images with 3D effects, which not only meet the requirements of wide field of view, high resolution, but also better meet the needs of users. Embodiment 4 The embodiment of the present invention provides a terminal, as shown in FIG. 6, specifically:

An image obtaining module 610, configured to acquire two images with overlapping regions;

The image matching module 620 is configured to register the two images according to the overlapping area, and synthesize the two images after registration.

Specifically, the image matching module 620 extracts feature points of two images, and extracts matching feature pairs of the two images in the feature points, and uses the matched feature pairs as alignment points to match the two images. quasi.

The feature points in the image blending module 620 may be any geometric or grayscale features suitable for image blending based on image properties. Embodiments of the present invention preferably use corner points as feature points to be extracted.

In this embodiment, the image matching module 620 can perform corner extraction by the following corner detection algorithm: Moravec operator corner detection, Forstner operator corner detection, Susan detection algorithm, and Plessy corner detection algorithm. Among them, the Plessy corner detection algorithm has excellent performance in terms of consistency and validity, and the extracted corner points are proved to have the advantages of rotation, translation invariance and stability.

In this embodiment, the improved Plessy corner detection algorithm is preferably used for corner extraction. At this time, the image matching module 620 includes: a calculation submodule 621, a setting submodule 622, and a screening submodule. 623 and extraction sub-module 624; wherein:

The calculation sub-module 621 is configured to convolve the image with a 3 x 3 convolution kernel for each image, obtain a partial derivative of each pixel of the image, and use the partial derivative to calculate the Plessy corner detection corresponding to each pixel point. a symmetric matrix M in the algorithm;

Setting a sub-module 622 for setting a selection window and a feature point available evaluation function R; wherein

R = ^Det(M , , where Z3⁄4t(M) = l^, Trace(M) = + ₂ , . are respectively matrix Μ Trace(M) + ε

The feature value, £· is a minimum value that makes the denominator non-zero; the screening sub-module 623 is configured to select a detection area on the image according to the selection window, and select the largest R value in the detection area. Pixels, move the selection window until the entire image is filtered;

The extraction sub-module 624 is configured to set a feature point determination threshold, and set, in the selected pixel points, a pixel point whose R value is greater than the determination threshold value as the extracted feature point.

Preferably, the boundary feature points in the image are deleted using a preset boundary template before the feature points of the modules 621 to 624 are extracted. After the feature points of the modules 621 to 624 are extracted, the sub-pixel feature points in each feature point are extracted, and the extracted sub-pixel feature points are used as the final extracted feature points.

In this embodiment, the image matching module 620 extracts matching feature pairs, including but not limited to: Hausdorff distance method, slack mark method, deterministic annealing algorithm, and iterative closest point algorithm (ICP). In this embodiment, in order to accurately extract the correct matching feature point pairs while removing the redundant feature points, preferably, the matching feature pairs are extracted by using the bidirectional maximum correlation coefficient method and the random sampling method. At this time, the image matching module 620 includes: a coarse matching sub-module 625 and an exact matching sub-module 626; wherein:

The coarse matching sub-module 625 is configured to perform coarse matching on the feature points in the two images by using the coarse matching bidirectional maximum correlation coefficient BGCC algorithm;

Preferably, before the coarse matching sub-module 625 performs coarse matching on the feature points in the two images, the median filter is used to smooth the two images, and the result of subtracting the original image from the filtered image is used as a rough match. The manipulated object of the process.

The exact matching sub-module 626 is configured to accurately match the matching feature pairs obtained by the rough matching by using a random RANSAC algorithm to obtain an accurately extracted matching feature pair. In this embodiment, the image blending module 620, preferably using an improved progressive fade-out synthesis method, sets the gray value ^χ of each pixel of the two images after registration to realize image synthesis; , setting

f(x,y) =

Where _ ;(x, , / ₂ (x, _y) respectively represent the gray values of the pixels in the two images, 4, e (0, l), and 4+ = i, respectively representing the asymptotic of the two images Factor, ifo ₀ r is a pre-set decision threshold, ;

/ ₂ denotes two images respectively. Furthermore, in the image synthesis, if the overlap area of the selected seam is too large, problems such as image blurring and inconspicuous edge information may occur. If the selected seam overlap area is too small, the seam stitching phenomenon of the image cannot be eliminated. Therefore, in the embodiment, for the processed image, the 7x7 region around the seam is used as the seam processing region, and the pixel in the seam region is linearly filtered by the template of 3x3, and the effect is best.

Embodiment 5

The embodiment provides a terminal. The embodiment includes all the functional modules in the fourth embodiment, and is an extension of the solution in the fourth embodiment. As shown in FIG. 7, the method includes:

An image obtaining module 710, configured to acquire two images having overlapping regions;

The image pre-processing module 720 is configured to process the two images acquired by the image obtaining module 710 according to the set pre-processing operation; wherein the pre-processing operation includes one or more of the following operations: verifying the acquired image, The two images are converted into the same coordinate system, the two images are smoothed and processed, and the initial positioning is performed to obtain a substantially overlapping region, and the overlapping region is taken as an extraction region of the feature point;

The image matching module 730 is configured to register the two images according to the overlapping area, and synthesize the two images after registration; specifically, the image matching module 730 extracts feature points of the two images, and The matching feature pairs of the two images are extracted from the feature points, and the matching image pairs are used as alignment points to register the two images. The 3D image generation module 740 is configured to acquire a composite image and another image having an overlapping area with the composite image, trigger the image combining module 730 to perform re-synthesis, and repeat the image capturing and image combining process to obtain a 3D image having a depth of field. Wherein, another image having a certain overlapping area with the composite image may be a non-composite image or a composite image.

In summary, the terminal in the embodiment of the present invention extracts feature point parameters directly for each image by taking two sets of images with different angles but overlapping areas, and then determines the degree of matching between the images according to each feature point, and rejects the error. The matching pair is combined and processed according to the image after registration, and a wide-field, high-resolution image is obtained, which greatly improves the user experience;

Furthermore, the terminal in the embodiment of the present invention performs spatial overlapping processing on a series of images by taking a series of images with different angles but overlapping regions to form a wide viewing angle scene including each image sequence information. The high-definition new image with 3D effect not only meets the requirements of wide field of view and high resolution, but also better meets the user's needs. The spirit and scope of the invention. Thus, it is intended that the present invention cover the modifications and variations of the inventions

Industrial applicability

According to the terminal and method of the embodiment of the present invention, the series of images are spatially overlapped by taking a series of images with different angles but overlapping regions to form a wide viewing angle scene including each image sequence information. High-definition new images with 3D effects, which not only meet the requirements of wide field of view and high resolution, but also better meet the needs of users.

Claims

Claim

A method for implementing image processing by a terminal, comprising:

Image acquisition step: acquiring two images with overlapping regions;

Image matching step: According to the overlapping area, two images are registered, and the two images after registration are combined.

2. The method according to claim 1, wherein the registering the two images according to the overlapping area comprises:

Extracting feature points of the two images, and extracting matching feature pairs of the two images in the feature points, and registering the two images with the matching feature pairs as alignment points.

3. The method of claim 2, wherein the feature points comprise corner points of an image.

4. The method according to claim 3, wherein the extracting feature points of the two images comprises: for each image, convolving with the image using a 3 x 3 convolution kernel to obtain pixel points of the image a partial derivative, and using the partial derivative to calculate a symmetric matrix M in a Plessy corner detection algorithm corresponding to each pixel point;

Trace(M) + ε Det(M, = λ, . Trace(M) = + λ, , , are the eigenvalues of the matrix Μ, respectively, ε is the minimum value that makes the denominator non-zero; Selecting a detection area on the image, filtering a pixel point having the largest R value in the detection area, and moving the selection window until the complete image is filtered;

The method according to claim 2 or 3 or 4, wherein the method further comprises: deleting the boundary feature points in the image by using a preset boundary template before extracting the feature points;

The method according to claim 2 or 3 or 4, wherein the matching feature pairs of the two images are extracted from the feature points, including: By using the coarse matching bidirectional maximum correlation coefficient BGCC algorithm, the feature points in the two images are coarsely matched, and the matching feature pairs obtained by the rough matching are accurately matched by the random sample RANSAC algorithm to obtain the accurately extracted matching feature pairs.

7. The method according to claim 6, wherein before the coarse matching of the feature points in the two images, the method further comprises:

The method according to any one of claims 1 to 4, wherein the synthesizing the two images after registration comprises: according to the progressive progressive synthesis method, each of the two images after registration The gray value of the pixel / (x, _y) is set; wherein, the setting rule includes:

Where , Λ (representing the gray value of the pixel in the two images, 4, ^ (0, 1), and 4 +^ = 1 respectively, indicating the progressive factor of the two images, ifoor is the preset judgment The threshold, _;, / ₂ represent two images.

9. The method of claim 8, wherein the synthesizing the two images after registration further comprises:

The method according to any one of claims 1 to 4, further comprising: between the image capturing step and the image combining step, further comprising:

Image pre-processing step: processing the two images acquired by the image obtaining step according to a set pre-processing operation; wherein, the pre-processing operation comprises one or more of the following operations: verifying the acquired image, and The image is converted to the same coordinate system, the two images are smoothed and processed, and the initial positioning is performed to obtain a rough estimated overlap region.

The method according to any one of claims 1 to 4, wherein the method further comprises: The 3D image generating step: acquiring a composite image and another image having an overlapping area with the composite image, performing the image combining step to perform re-synthesis, repeating the image capturing and image combining process, and obtaining a 3D image having a depth of field.

12. A terminal, comprising:

The terminal according to claim 12, wherein the image matching module is configured to extract feature points of two images, and extract matching feature pairs of two images in the feature points, to match the The feature pair is the alignment point, and the two images are registered.

14. The terminal of claim 13, wherein in the image combining module, the feature point includes a corner point of the image.

The terminal according to claim 14, wherein the image matching module further comprises: a calculation sub-module configured to convolve with the image by using a 3 x 3 convolution kernel for each image to obtain an image a partial derivative of the pixel, and using the partial derivative to calculate a symmetric matrix M in the Plessy corner detection algorithm corresponding to each pixel;

Setting a sub-module, which is set to set a selection window, and a feature point available evaluation function R; wherein, _{R =} ^Det(M , , Det{M) = l ₁ , Trace{M) = + l ₁ , , respectively

Trace(M) + ε

The characteristic value of Μ, £· is a minimum value that makes the denominator non-zero; the screening sub-module is arranged to select a detection area on the image according to the selection window, and the R value is selected to be the largest in the detection area. Pixels, move the selection window until the full image is filtered;

The terminal according to claim 13 or 14 or 15, wherein the image matching module further comprises:

a coarse matching sub-module configured to coarsely match feature points in the two images using a coarse matching bidirectional maximum correlation coefficient BGCC algorithm;

The terminal according to any one of claims 12 to 15, wherein the image matching module is further configured to: according to the progressive fade-out synthesis method, the gray value of each pixel of the two images after registration/ (χ, set; where the setting rules include:

f(x, y) =

Where , Λ (representing the gray value of the pixel in the two images, 4, ^ (ο, ι), and 4 + ^ = 1 respectively, indicating the progressive factor of the two images, ifo ₀ r is preset The judgment threshold, ;, / ₂ respectively represent two images.

The terminal according to any one of claims 12 to 15, wherein the terminal further comprises: an image preprocessing module, and/or a 3D image generating module;

The image pre-processing module is configured to process the two images acquired by the image acquisition module according to a set pre-processing operation; wherein the pre-processing operation comprises one or more of the following operations: Image, converting the two images to the same coordinate system, smoothing and filtering the two images, and initially locating to obtain a rough estimated overlap region;

The 3D image generation module is configured to acquire a composite image and another image having an overlapping area with the composite image, trigger the image matching module to perform re-synthesis, repeat image acquisition and image matching process, and obtain a depth of field 3D image.