CN116309055A

CN116309055A - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN116309055A
Application number: CN202310220609.9A
Authority: CN
Inventors: 鲍捷; 左成文; 请求不公布姓名
Original assignee: Quantaeye Beijing Technology Co ltd; Tsinghua University
Current assignee: Quantaeye Beijing Technology Co ltd; Tsinghua University
Priority date: 2023-03-08
Filing date: 2023-03-08
Publication date: 2023-06-23

Abstract

The present disclosure relates to a method and apparatus for processing an image, an electronic device, and a storage medium, where the method for processing an image includes: acquiring at least two images to be spliced; determining image characteristic points corresponding to each image to be spliced; performing similarity matching on image feature points in different images to be spliced, and constructing a matching point set according to the successfully matched image feature points among the different images to be spliced; amplifying the image characteristic points of the matching point set; according to the amplified matching point set, adjusting the warping degree of at least one image to be spliced; and performing image stitching on the at least two adjusted images to be stitched to obtain a composite image. The implementation of the method is beneficial to improving the synthesis effect of the images to be spliced.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of information processing, and in particular relates to an image processing method, an image processing device, electronic equipment and a storage medium.

Background

Image stitching is widely studied and applied in the field of computer vision. In fields such as environmental monitoring, remote sensing imaging, automatic driving and the like, the view angle range of a single sensor is limited, and a plurality of images acquired by the single sensor are spliced to obtain images with larger view angles.

However, in the process of image stitching, artifacts, holes and the like usually occur. In the related art, the splicing effect of the images is poor, so how to improve the splicing effect of different images is a technical problem which needs to be solved by a developer.

Disclosure of Invention

The disclosure provides an image processing technical scheme.

According to an aspect of the present disclosure, there is provided a processing method of an image, the processing method including: acquiring at least two images to be spliced; determining image characteristic points corresponding to each image to be spliced; performing similarity matching on image feature points in different images to be spliced, and constructing a matching point set according to the successfully matched image feature points among the different images to be spliced; amplifying the image characteristic points of the matching point set; according to the amplified matching point set, adjusting the warping degree of at least one image to be spliced; and performing image stitching on the at least two adjusted images to be stitched to obtain a composite image.

In a possible implementation manner, the amplifying the image feature points by using the matching point set includes: determining a reference image and a target image in the at least two images to be spliced, and determining a first corresponding relation of image feature points between the reference image and the target image according to the matching point set; wherein the target image is subjected to image adjustment by taking a reference image as a datum; the first corresponding relation is established based on a plurality of groups of image characteristic points in the matching point set; and amplifying the image characteristic points of the matching point set according to the first corresponding relation, the pixel points in the reference image and the pixel points in the target image.

In a possible implementation manner, the amplifying the image feature points of the matching point set according to the first correspondence, the pixel points in the reference image, and the pixel points in the target image includes: the following steps are executed until a preset condition is reached: dividing the target image into a plurality of target sub-images; taking the pixel point with the minimum pixel gradient in each target sub-image as the target pixel point of each target sub-image; the pixel gradient is used for representing the variation degree of pixel values among pixel points; determining a corresponding reference pixel point of each target pixel point in a reference image according to the first corresponding relation; taking the pixel point which is in the preset range of each reference pixel point, has the highest target pixel point similarity corresponding to each reference pixel point and is higher than a preset similarity threshold value as a new reference pixel point corresponding to each target pixel point, and adding each new reference pixel point and the target pixel point corresponding to each new reference pixel point as a group of successfully matched image feature points into the matching point set; and aiming at the target sub-image corresponding to each new reference pixel point, taking the target sub-image as a new target image, and re-executing the division of the target image into a plurality of target sub-images.

In a possible implementation manner, the determining, according to the first correspondence, a reference pixel point corresponding to each target pixel point in a reference image includes: determining a weight matrix corresponding to each target pixel point according to the distance between the target pixel point and each image characteristic point in the target image; and determining a corresponding reference pixel point of each target pixel point in the reference image according to the weight matrix, the first corresponding relation and the position of each target pixel point in the target image.

In a possible implementation manner, after the image feature point amplification is performed on the matching point set, the processing method further includes: establishing a second corresponding relation according to the amplified multiple groups of image characteristic points in the matching point set; determining the mapping error of each group of image characteristic points in the amplified matching point set according to the second corresponding relation; screening the image characteristic points in the amplified matching point set according to the mapping error; the step of adjusting the warping degree of at least one image to be spliced according to the amplified matching point set comprises the following steps: and adjusting the warping degree of at least one image to be spliced according to the screened and amplified matching point set.

In a possible implementation manner, the adjusting the warping degree of at least one image to be spliced according to the screened and amplified matching point set includes: dividing the target image into a plurality of pixel blocks; the difference value between the feature vectors corresponding to the pixel features of different pixel points in each pixel block is smaller than a preset difference value; and adjusting the warping degree corresponding to each pixel block according to the second corresponding relation and the screened and amplified matching point set.

In a possible implementation manner, the adjusting the warpage level corresponding to each pixel block according to the second correspondence and the filtered and amplified matching point set includes: determining a third corresponding relation corresponding to each pixel block according to the screened and amplified image characteristic points in the matching point set; wherein the third correspondence is used to represent at least one of scaling, rotation, offset between the target image and the reference image; determining a rotation angle corresponding to the target image according to the second corresponding relation; rotating the target image according to the rotation angle; determining the maximum projection distance and the minimum projection distance of a plurality of block center points corresponding to the pixel blocks on a horizontal axis after the target image rotates; determining a third corresponding relation weight and a second corresponding relation weight corresponding to each pixel block according to the maximum projection distance and the minimum projection distance; and determining the warping degree corresponding to each pixel block according to the second corresponding relation, the second corresponding relation weight, the third corresponding relation and the third corresponding relation weight corresponding to each pixel block.

In a possible implementation manner, the adjusting the warpage level corresponding to each pixel block according to the second correspondence and the filtered and amplified matching point set further includes: dividing the reference image into a plurality of pixel blocks; and adjusting the warping degree corresponding to each pixel block in the reference image according to the warping degree corresponding to each pixel block in the target image.

In a possible implementation manner, the determining the image feature point corresponding to each image to be stitched includes: generating a first image corresponding to each image to be spliced in each scale space which is sequentially arranged aiming at each image to be spliced; the scale space is used for adjusting the image size of the images to be spliced; the arrangement sequence of the scale space is inversely related to the image reduction coefficient corresponding to the scale space; for each first image, generating a plurality of second images with different scale parameters in the same scale space according to a plurality of scale parameters which are sequentially arranged; the scale parameter is used for adjusting the resolution of the first image; the arrangement sequence of the scale parameters is inversely related to the corresponding fuzzy degree of the scale parameters; enhancing the image texture of each second image; aiming at each image to be spliced, generating a differential image corresponding to each image to be spliced in each scale space according to the adjacent enhanced second images of each image to be spliced in the same scale space; and screening pixel points in the differential images, and taking the screened pixel points as image characteristic points corresponding to each image to be spliced.

In a possible implementation manner, the filtering pixel points in the differential image includes: comparing each pixel point with adjacent pixel points in the differential image, first pixel points at corresponding positions in the adjacent scale space and adjacent pixel points in the adjacent scale space, and determining extreme points in each pixel point; wherein the extreme point is a minimum pixel value point or a maximum pixel value point; and taking the extreme point as the pixel point after screening under the condition that the position corresponding to the extreme point is positioned in a non-edge area in the differential image or the pixel value corresponding to the extreme point is higher than a preset value.

According to an aspect of the present disclosure, there is provided an image processing apparatus including: the image acquisition module is used for acquiring at least two images to be spliced; the characteristic point determining module is used for determining image characteristic points corresponding to each image to be spliced; the matching point set construction module is used for carrying out similarity matching on the image characteristic points in different images to be spliced and constructing a matching point set according to the successfully matched image characteristic points among the different images to be spliced; the matching point set amplifying module is used for amplifying the image characteristic points of the matching point set; the warping degree adjusting module is used for adjusting the warping degree of at least one image to be spliced according to the amplified matching point set; and the synthetic image generation module is used for carrying out image stitching on the at least two adjusted images to be stitched to obtain a synthetic image.

According to an aspect of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of processing an image as described above.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described image processing method.

In the embodiment of the disclosure, at least two images to be spliced can be obtained, image feature points corresponding to each image to be spliced are determined, then similarity matching is carried out on the image feature points in different images to be spliced, a matching point set is constructed according to the successfully matched image feature points among the different images to be spliced, then the image feature points of the matching point set are amplified, then the warping degree of at least one image to be spliced is adjusted according to the amplified matching point set, and finally image splicing is carried out on the adjusted at least two images to be spliced to obtain a synthetic image. The image processing method provided by the embodiment of the disclosure can amplify the matched image characteristic points between the images to be spliced, so that the splicing relevance between the images to be spliced is improved, and the synthesis effect of the images to be spliced is improved. In addition, the embodiment of the disclosure adjusts the warping degree of the images to be spliced according to the amplified matching point set, so that the problem of relevance loss of the images to be spliced caused by different shooting angles can be reduced, and the synthesis effect of the images to be spliced can be improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the technical aspects of the disclosure.

Fig. 1 shows a flowchart of a method of processing an image provided according to an embodiment of the present disclosure.

Fig. 2 shows a flowchart of a method of processing an image provided according to an embodiment of the present disclosure.

Fig. 3 shows a reference schematic of fractional order differential order provided in accordance with an embodiment of the present disclosure.

Fig. 4 shows a reference schematic diagram of a method of processing an image provided according to an embodiment of the present disclosure.

Fig. 5 shows a reference schematic diagram of a gaussian pyramid, a gaussian differential pyramid, provided in accordance with an embodiment of the present disclosure.

Fig. 6 shows a reference schematic diagram of a mask window provided in accordance with an embodiment of the present disclosure.

Fig. 7 shows a flowchart of a method of processing an image provided according to an embodiment of the present disclosure.

Fig. 8 shows a reference schematic diagram of image feature point augmentation provided in accordance with an embodiment of the present disclosure.

Fig. 9 illustrates a reference schematic diagram for adjusting a warp degree of a target image provided according to an embodiment of the present disclosure.

Fig. 10 illustrates a reference schematic diagram for adjusting a warp degree of a target image provided according to an embodiment of the present disclosure.

Fig. 11 shows a block diagram of an image processing apparatus provided according to an embodiment of the present disclosure.

Fig. 12 shows a block diagram of an electronic device provided in accordance with an embodiment of the present disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order not to obscure the present disclosure.

In the related art, the determination of the overlapping area of the images to be spliced is generally performed by determining the image feature points in the images to be spliced, and then overlapping the overlapping areas in the images to be spliced, so that the composite image can be obtained. This has the following problems: 1. the registration between the images to be spliced is easy to have a problem, namely the generation accuracy of the overlapping area is low. 2. The shooting angles of the images to be spliced may be different, so that the synthesis between the images to be spliced is difficult, or the number of artifacts or holes in the synthesized images is large.

In view of this, an embodiment of the present disclosure provides a method for processing an image, where at least two images to be stitched may be obtained, and image feature points corresponding to each image to be stitched are determined, then similarity matching is performed on the image feature points in different images to be stitched, a matching point set is constructed according to image feature points successfully matched between different images to be stitched, then image feature point amplification is performed on the matching point set, and then a warping degree of at least one image to be stitched is adjusted according to the amplified matching point set, and finally image stitching is performed on the at least two adjusted images to be stitched, so as to obtain a composite image. The image processing method provided by the embodiment of the disclosure can amplify the matched image characteristic points between the images to be spliced, so that the splicing relevance between the images to be spliced is improved, and the synthesis effect of the images to be spliced is improved. In addition, the embodiment of the disclosure adjusts the warping degree of the images to be spliced according to the amplified matching point set, so that the problem of relevance loss of the images to be spliced caused by different shooting angles can be reduced, and the synthesis effect of the images to be spliced can be improved.

Referring to fig. 1, fig. 1 illustrates a flowchart of a method of processing an image provided according to an embodiment of the present disclosure. As shown in fig. 1, the processing method includes:

and step S100, acquiring at least two images to be spliced. The image to be stitched may be image data in any data format in the related art, for example: JPG files, PNG files (JPG, PNG are data formats of image files) and the like in the related art. Some identical target objects may exist between different images to be stitched, where the images to be stitched may be selected by a user, and embodiments of the present disclosure are not limited herein.

Step S200, determining image feature points corresponding to each image to be spliced. Illustratively, the above image feature points may be extracted by SIFT algorithm (Scale-invariant feature transform, scale-invariant feature transform algorithm), SURF algorithm (Speeded Up Robust Features, accelerated document feature algorithm), ORB algorithm (Oriented Fast and Rotated Brief, one of algorithms based on feature detection algorithm and vector creation algorithm) or the like in the related art, or by machine learning model constructed based on the above algorithm ideas, the embodiments of the present disclosure are not limited herein. The image feature points can be used for representing the image features of the images to be spliced. The above-described image feature points may be constructed by position information, angle information, descriptor information, and the like, for example.

Referring to fig. 2, fig. 2 is a flowchart illustrating a method for processing an image according to an embodiment of the disclosure, and in conjunction with fig. 2, in a possible implementation manner, step S200 may include: step S210, generating a first image corresponding to each image to be spliced in each scale space which is sequentially arranged according to each image to be spliced. The scale space is used for adjusting the image size of the images to be spliced, and the arrangement sequence of the scale space is inversely related to the image reduction coefficient corresponding to the scale space. Illustratively, taking a gray scale image as an example herein, for gray scale image I (x, y), the definition of the scale space may be: l (x, y, σ) =g (x, y, σ) ×i (x, y), where x, y are coordinates in horizontal and vertical directions in a two-dimensional space of a certain pixel point in the image to be stitched. Gaussian function

* Representing convolution operation, sigma is a scale parameter, and determines the smoothness of an image, and the larger the sigma is, the coarser the scale is, namely the lower the resolution is, and the higher the Gaussian blur degree is. e is a mathematically fixed constant of about 2.71. Illustratively, adjacent scale spacesThe zoom magnification of the image may be the same, for example: if the scale spaces 1, 2, 3 are adjacent (for brevity, only three layers are used for illustration), the reduction coefficient of the scale space 1 is larger than the reduction coefficient of the scale space 3 (which can be regarded as that the height of the scale space 1 in the gaussian pyramid is higher than that of the scale space 3), the image reduction coefficient of the scale space 3 can be set to 1, that is, when the image to be spliced is located in the scale space 3, the image to be spliced is not subjected to reduction processing. The image reduction coefficient of the scale space 2 can be set to be 2, that is, when the image to be spliced is positioned in the scale space 2, the length and the width of the image to be spliced are reduced to be 1/2 of the original length and the width. On this basis, the reduction coefficient of the scale space 1 can be set to be 4 (2 x 2), that is, when the image to be spliced is located in the scale space 1, the length and the width of the image to be spliced are reduced to be 1/4 of the original (compared with the scale space 3). In this example, the scaling factor between the scale space 1 and the scale space 2 is 2, which is the same as the scaling factor between the scale space 2 and the scale space 3, that is, the scaling factor of the image may be the same between the adjacent scale spaces. In other examples, the zoom magnification may be flexibly set, and embodiments of the present disclosure are not limited herein.

Step S220, for each first image, generating a plurality of second images with different scale parameters in the same scale space according to a plurality of scale parameters arranged in sequence. The scale parameters are used for adjusting the resolution of the first image, and the arrangement sequence of the scale parameters is inversely related to the blurring degree corresponding to the scale parameters. For example, σ may be directly changed in the above-described scale space to change the resolution of the images to be stitched in the same scale space.

In step S230, the image texture of each second image is enhanced. For example, a Riemann-LiuViel fractional derivative operation may be performed on the scale space to enhance the image texture of each second image. A mask window function R (x, y) may be defined as:

wherein the subscript of R represents the distance from the center of the mask window (the distance is represented by the number of panes here), and n is the center point of the mask window divided byThe total number of layers (e.g., the total number of layers of the mask window of 3*3 size is 1), γ is an intensity factor, typically a positive integer, Γ (·) is a Γ function (or gamma function) in the related art, P is a fractional order differential order, typically a fraction between 0 and 1 (as shown in connection with fig. 3, fig. 3 shows a reference schematic diagram of the fractional order differential order provided according to an embodiment of the present disclosure). Convolving the mask window with the scale space to obtain L ^′ R (x, y) L (x, y, σ). For example: processing can be performed through a mask window of 3x3 size, γ can be set to 5, P to 0.5, R _i Is R ₁ . It should be appreciated that the image texture may also be enhanced by the texture enhancement method in the related art, for example: histogram equalization, gamma transformation, laplace transformation, etc., embodiments of the present disclosure are not limited herein.

Step S240, for each image to be stitched, generating a differential image corresponding to each image to be stitched in each scale space according to the second images adjacent and enhanced in the same scale space. For example, pixel difference calculation may be performed on adjacent second images in the same scale space, so as to obtain the differential image. The above-described difference may be expressed as: d (x, y, σ) =l ^′ (x,y,kσ)-L ^′ (x, y, σ), where k is a coefficient representing a different σ, can be flexibly set by a developer. The above differential image includes pixel value differences between the images to be stitched under different scale parameters, which can be regarded as image features of the differential image under different scale parameters (or different resolutions).

Step S250, screening pixel points in the differential image, and taking the screened pixel points as image feature points corresponding to each image to be spliced. For example, a pixel threshold may be screened, i.e., pixels above the threshold are considered to be screened pixels. The developer may set the pixel threshold according to the actual situation, and the embodiments of the disclosure are not limited herein.

In one possible implementation, step S250 may include the steps of: and comparing each pixel point with adjacent pixel points in the differential image, a first pixel point at a corresponding position in the adjacent scale space and adjacent pixel points in the adjacent scale space, and determining an extreme point in each pixel point. Wherein the extreme point is a minimum pixel value point or a maximum pixel value point. Illustratively, the corresponding location may be a pixel coordinate obtained via a coordinate transformation (e.g., multiplying or dividing the coordinate value of the pixel by the zoom magnification above) in the image in which the scale parameters are the same in the adjacent scale space. And taking the extreme point as the pixel point after screening under the condition that the position corresponding to the extreme point is positioned in a non-edge area in the differential image or the pixel value corresponding to the extreme point is higher than a preset value. Illustratively, the foregoing preset values may be flexibly set by a developer, and embodiments of the present disclosure are not limited herein. According to the embodiment of the disclosure, the pixel points in the differential image can be screened in a mode of screening the extreme points, and the extreme points can be regarded as image feature points which can best represent image features under different scale parameters. Referring to fig. 4 in combination with fig. 4, fig. 4 shows a reference schematic diagram of an image processing method according to an embodiment of the disclosure, and as shown in fig. 4, in the embodiment of the disclosure, compared with the SIFT, SURF, ORB, KAZE, AKAZE algorithm, the distribution of image feature points is more and more reasonable.

Referring to fig. 5 and 6, fig. 5 shows a reference schematic diagram of a gaussian pyramid and a gaussian differential pyramid provided according to an embodiment of the present disclosure, and fig. 6 shows a reference schematic diagram of a mask window provided according to an embodiment of the present disclosure. As shown in fig. 5, the triangle on the left side in fig. 5 is a gaussian pyramid, the triangle on the right side is a gaussian differential pyramid, and the window of 3*3 in fig. 6 is a mask window. The embodiments of the disclosure provide an example for reference, in which a gaussian pyramid of an image to be stitched may be constructed first, where the gaussian pyramid has a total of O layers (O may be any positive integer), and the length and width of the image to be stitched corresponding to the upper layer are half of those of the next layer, and each layer may be defined by s+3 (where the minimum S may be 0, and the minimum number of images of each layer is 3 in consideration of the accuracy of subsequent operations, which may be set according to the actual situation), the image (the number of images andthe number of scale parameters is positively correlated), the O-th layer, the scale parameters of the s-th image may be

Where S represents the S-th image, S represents the value (which can be set by the developer according to the actual situation and remains the same when obtaining the different scale parameters), and σ ₀ For the initial scale, each image is subjected to fractional differentiation. And then constructing a Gaussian differential pyramid, wherein the Gaussian differential pyramid is obtained by differentiating two similar images of each layer of the Gaussian pyramid, and the Gaussian differential pyramid has O layers, and each layer has S+2 differential images. And then searching for extreme points in the differential image. For example, for each pixel in the differential image, the pixel value between the pixel and 8 surrounding pixels, and 9 pixels in the upper and lower adjacent images in the differential pyramid (for simplicity of operation, the pixel value may be set to a gray value), and if the pixel value is larger or smaller than the pixel value of all the 26 pixels, the pixel may be considered as an extremum point. And then the contrast threshold value (i.e. the preset value in the above) in the extreme points and the edge points can be removed, and the rest points are regarded as the filtered pixel points. Finally, a search range is determined according to the scale of the feature point (the scale can be defined as the scale parameter and the scale space in which the feature point is found), the pixel gradient (used for representing the variation degree of the pixel value) in the range is calculated, and the corresponding gradient direction is used as the main direction of the feature point. And then counting the pixel gradient directions within a certain range of the feature points, and rotating to the main direction, thereby constructing a direction histogram as a descriptor vector of the feature points, wherein the descriptor vector can be used for representing the image features of the feature points, and the generation process of the descriptor vector can be specifically referred to the related technology.

With continued reference to fig. 1, step S300 performs similarity matching on image feature points in different images to be stitched, and constructs a matching point set according to image feature points successfully matched between the different images to be stitched. Illustratively, the similarity matching may be obtained by measuring the vector distance of descriptors between different image feature points (e.g., using the descriptor vectors described above), the closer the vector distance of the descriptors, the more similar the different image feature points are. It should be understood that the similarity may be measured by any method in the related art, and embodiments of the disclosure are not described herein. In one example, a developer may set a similarity threshold, and in the case where the similarity between the plurality of image feature points is higher than the similarity threshold, the plurality of image feature points may be considered to be successfully matched. For example, in the case that the matching of the plurality of image feature points is successful, the plurality of image feature points may be stored as a set of feature points in the matching point set, that is, each set of image feature points in the matching point set may be a set of successfully matched image feature points.

And step S400, amplifying the image characteristic points of the matching point set. In one example, the amplification of image feature points may be determined by, for example: determining a group of image characteristic points A, B in the matching point set, then determining adjacent pixel points (or pixel points within a certain distance) of the point A in the image to be spliced as a point set a, and adjacent pixel points (or pixel points within a certain distance) of the point B in the image to be spliced as a point set B, performing similarity matching on the pixel points in the point set a and the pixel points in the point set B, and adding each group of successfully matched pixel points into the matching point set. It should be appreciated that image stitching may not be limited to stitching of two images, but that multiple images may be stitched, for example: there are common parts in the three or four images to be spliced, and the total number of image feature points in each group of image feature points in the matching point set can be 3 or 4. For example; the three images to be spliced have a common part, 100 groups of image feature points are successfully matched, and each group of image feature points can comprise 3 image feature points, namely each image to be spliced corresponds to one image feature point in each group of image feature points.

Referring to fig. 7, fig. 7 is a flowchart illustrating a method of processing an image according to an embodiment of the present disclosure. In one possible implementation, step S400 may include: step S410, determining a reference image and a target image in the at least two images to be stitched, and determining a first correspondence of image feature points between the reference image and the target image according to the matching point set. And the target image is subjected to image adjustment by taking a reference image as a standard, and the first corresponding relation is established based on a plurality of groups of image characteristic points in the matching point set.

Embodiments of the present disclosure are presented herein as an example for reference, with reference and target images respectively denoted as I _r And I _t Assume that the matching point pair (i.e., the set of matching points described above) is represented as { p } _i ,q _i I=0, 1, …, M-1, where M represents the total number of matched pairs, p _i The homogeneous coordinates of (a) can be expressed as [ x ] _i ,y _i ,1] ^T ，q _i Can be expressed as [ x 'i, y ]' _i ,1] ^T T represents a matrix transpose whose correspondence may be represented by homography matrix H (i.e., the first correspondence above) as p _i ＝Hq _i Substituting the above-mentioned coordinate values,

let h= [ h ] ₁ ,h ₂ ,…,h ₉ ] ^T Constraint h ² =1, at p _i ＝Hq _i On the basis of cross multiplication p _i 0=p can be obtained _i ×Hq _i Further, 0= can be derived>

Let matrix

a _i Is linearly independent of the two row vectors in (a). All feature points (i=0, 1, …, N) correspond to matrix a _i Is denoted as A, wherein +.>

Solving the optimal h-transform into solving the least square problem:

exemplary embodimentsGround, the above a _i A matrix of size 2 rows and 9 columns ()>

3 columns each), then N a _i Can be stacked into a matrix a of size 2N rows and 9 columns. When the number of the matching point pairs is greater than 4 and the matching point pairs are not collinear, the number of linear irrelevant vectors in A is far greater than 8, singular Value Decomposition (SVD) can be used for solving the linear equation set Ah=0, and after the singular value decomposition of A, the optimal solution is the eigenvector corresponding to the minimum eigenvalue. Illustratively, there are 9 unknown parameters in h, i.e., h ₁ To h ₉ In the constraint h ² In the case of =1, then 8 parameters are also required to solve, and 4 sets of matching points can determine 8 equations (4 on the abscissa and 4 on the ordinate), so it can be deduced that at least 4 sets of matching points are required to solve for the h vector. The number of the matching point pairs may be set to 4 or more.

Step S420, performing image feature point amplification on the matching point set according to the first correspondence, the pixel points in the reference image, and the pixel points in the target image. For example, the pixels in the same direction and distance as the pixels in the matching point set in the reference image and the target image may be added as a group of pixels to the matching point set. For example: if the point a in the reference image is successfully matched with the point b in the target image, the pixel point adjacent to the left side of the point a and the pixel point adjacent to the left side of the point b are added into the matched point set as a group of pixel points.

In one possible implementation, step S420 may include: the following steps are performed until a preset condition is reached, which may include, for example: at least one of the total number of the image characteristic points in the matching point set reaches the preset total number, the dividing times of the target image reaches the preset times, and the total number of the pixel points in the target sub-image reaches the preset pixel point number: the target image is divided into a plurality of target sub-images. By way of example, the target image may be segmented by any image segmentation algorithm in the related art, and embodiments of the present disclosure are not limited herein. And then taking the pixel point with the minimum pixel gradient in each target sub-image asAnd the pixel gradients are used for representing the variation degree of pixel values among the pixel points. According to the embodiment of the disclosure, the pixel points with the minimum gradient of the pixel values in the pixel points around the central pixel point of the target sub-image are used for replacing the central pixel point of the target sub-image, so that the occurrence probability of noise points can be reduced. And then, according to the first corresponding relation, determining a corresponding reference pixel point of each target pixel point in the reference image. Illustratively, this step may include: and determining a weight matrix corresponding to each target pixel point according to the distance between the target pixel point and each image characteristic point in the target image. And determining a corresponding reference pixel point of each target pixel point in the reference image according to the weight matrix, the first corresponding relation and the position of each target pixel point in the target image. For example, a corresponding to different characteristic points can be obtained by adopting a moving direct linear transformation mode in the related art _i Matrix assigns different weights w _i Weight w _i And the current point q _* Distance i-th feature point q _i The use of gaussian weights can be expressed as:

wherein I II are generally the euclidean distance is represented as, gamma is the bias constant introduced to prevent numerical problems. The Gaussian weights can be used for differentially representing the weighting weights of different characteristic points corresponding to different positions, but the tail decays rapidly, the weighting of the region far away from the characteristic points is too low, and the additional parameter gamma needs to be determined to prevent numerical problems. When gamma is too large, the ownership weight w _i Tend to be consistent, degenerate into global homographies; when γ is too small, "extrapolation artifacts" and "wave effects" are created in the non-overlapping region. To eliminate this effect, a Student's t weight with slower tail decay may be used instead of the gaussian weight: />

The Student's t weight tail decays more slowly, extrapolation artifacts and wave effects can be effectively reduced, and the numerical problem is solved without applying bias. Introducing a weight w _i After that, the least squares problem is +.>

Wherein W (q) _* )＝diag(w ₁ (q _* ),w ₁ (q _* ),…,w _N (q _* ),w _N (q _* ) And), wherein W (q _* ) Is a matrix of 2N x 2N, where the value of N is the number of groups of pixels (or a) _i Number of (d). The introduction of the weight can solve the local homographies of different positions (points) in the image, and the error caused by local distortion can be reduced. Better alignment can be achieved by locally homography. However, solving the local homographies of all pixels of the entire image is very time-consuming and costly, and in order to reduce the amount of computation, the disclosed embodiments divide the target graph into different grids, calculate the local homographies of the center pixels of each grid to guide the grid deformation, and the warping effect and the time complexity are related to the grid division. If A x B grids are divided on the target graph, the original grid center point is replaced by the pixel with the smallest gradient of pixels immediately adjacent to the grid center point (noise influence is prevented), and the result is marked as + >

b=0, 1, …, B-1, computing the local homography corresponding to the grid center point: />

Let->

Representation->

The matrix after remodeling was 3×3 in size. Grid point->

The corresponding point on the reference image is denoted +.>

Then it can be determined +.>

And then taking the pixel point which is in the preset range of each reference pixel point, has the highest target pixel point similarity corresponding to each reference pixel point and is higher than a preset similarity threshold value as a new reference pixel point corresponding to each target pixel point, and adding each new reference pixel point and the target pixel point corresponding to each new reference pixel point as a group of successfully matched image feature points into the matching point set. In one example, the dot pair +.>

Not necessarily true pixel correspondence, there may be a few pixel offsets from the true matching point location. The embodiment of the disclosure can judge the real corresponding relation through the feature descriptor sub-vector corresponding to the image feature points. The feature descriptors are used as vector representations of the neighborhood pixel gradient histograms of the key points, are used for encoding information of local images near the key points, and have high identification. The presently disclosed embodiments provide an example herein for reference, let d (·) denote the descriptor generating function in the related art, S (p) = { p _(l) |||p _(l) -p| < epsilon } means that all sets of points with a distance to point p less than epsilon are +.>

Comparison of

And->

Is the distance between (2)And (5) separating. If->

Represented by->

No correct AND +.>

And (3) removing grid matching points from matched matching point pairs, otherwise updating the positions of the matching points in the reference map:

the values of epsilon and zeta are not limited in the embodiment of the disclosure, and can be flexibly set by a developer according to actual conditions. L represents the point->

And the total number of the pixel points in the preset range. Matching the updated grid with the point pairs +.>

The pixel points which are successfully matched are added into a matching point set, at the moment, the matching point set comprises a plurality of groups of pixel points formed by image feature points and a plurality of groups of amplified pixel points, the increase of the pixel points in the matching point set indicates that the reference image and the target image have a richer pixel point corresponding relation, and when the scene textures are weaker or different depth planes exist at the same time, the local homography estimation is more accurate, so that the subsequent warping adjustment is facilitated. And finally, aiming at the target sub-image corresponding to each new reference pixel point, taking the target sub-image as a new target image, and re-executing the division of the target image into a plurality of target sub-images. Illustratively, the A×B grids may be continuously divided within the successfully matched grids, and the above steps may be repeated to finally obtain a matching point set { p } _i ,q _i And thus arranged to increaseMore matching point pairs. Because the process introduces more successfully matched pixel points, the occurrence probability of phenomena such as shielding artifact, distortion void and the like of the finally obtained synthetic image can be reduced. Referring to fig. 8, fig. 8 shows a reference schematic diagram of image feature point amplification provided according to an embodiment of the present disclosure, where an image to be stitched on the left side is a reference image, an image to be stitched on the right side is a target image, and as the number of repeated steps increases, successfully matched pixels increase.

In a possible implementation manner, after step S400, the processing method may further include: and establishing a second corresponding relation according to the amplified multiple groups of image characteristic points in the matching point set. And then determining the mapping error of each group of image characteristic points in the amplified matching point set according to the second corresponding relation. And screening the image characteristic points in the amplified matching point set according to the mapping error. For example, since the number of matching point pairs (or a group of successfully matched pixel points) in the matching point set is large, the second correspondence may not accurately represent the correspondence of all the matching point pairs. Therefore, the embodiment of the disclosure sets a mapping error, and screens the image feature points in the amplified matching point set in a mapping error mode. The disclosed embodiments provide an example herein for reference: the posterior probability can be calculated by using a Bayes formula, and feature point screening is refined. Definition of delta _i ＝p _i -Hq _i ＝[Δx _i ,Δy _i ,0] ^T Representing the error between the projection vector of the pixel in the reference image and the pixel matched with the pixel in the target image (i.e. the mapping error above), delta = { delta _i And represents the set of error vectors for all matching point pairs in the set of matching points. Let the ith pair of the reference image and the target image match the point pair { p } _i ,q _i Whether or not (which can be defined as whether or not the correspondence between the two is objectively correct) is represented by a binary symbol τ _i Representing the probability p (τ _i ＝1)＝p ₁ ，p(τ _i ＝0)＝1-p ₁ Assume that the error Deltax or Deltay along the x or y direction between matching points obeys a zero-mean Gaussian distribution

Or->

And the feature points are spaced apart by a sufficient number of pixels. Let event A _i ＝{|Δx _i |>3σ _x or|Δy _i |>3σ _y And the error of the ith pair of matching points along the x or y direction is larger than 3 times of standard deviation. Then p (A) _i |τ _i =1) =2 (1- Φ (3)), where Φ (3) is about 0.998650, event a occurs _i The probability of (a) is p (a) _i )＝1-(2Φ(3)-1)p ₁ When the error of the ith pair of matching points meets the event A by Bayesian formula _i And is the probability of outliers->

Carry in the numerical value, when p ₁ At a value of p (. Tau.) of 0.9973 or less _i ＝0|A _i )>0.5, considered as outliers (or points that should be eliminated). If the matching point pair { p } _i ,q _i More than 0.0027N obeys A to matching points _i (0.0027 is 1-p ₁ Obtained), the probability of the matching point pairs being mismatching is not less than fifty percent (of course, the value can be set according to actual requirements as a preset value), and Bayesian refinement is performed after the matching point pairs are removed until the errors of all the matching point pairs are within the global homography constraint range. Illustratively, the second correspondence condition may not accurately match each set of matching point pairs, so even if the objective correspondence is correct, there is an error value satisfying A _i By combining the conditions, the embodiment of the disclosure can reduce the probability of mismatching the pixel points in the filtered matching point set. In this case, the adjusting, according to the amplified matching point set, the warping degree of at least one image to be stitched includes: and adjusting the warping degree of at least one image to be spliced according to the screened and amplified matching point set.

With continued reference to fig. 1, in step S500, the warping degree of at least one image to be stitched is adjusted according to the amplified matching point set. For example, the embodiment of the disclosure may warp all or part of the images to be stitched according to the amplified matching point set, so that the independent sense of the images to be stitched with different viewing angles after stitching is reduced, and the fusion degree of the images is further enhanced. Fig. 9 and 10 are reference diagrams showing the adjustment of the degree of warpage of the target image according to the embodiment of the present disclosure, as shown in fig. 9 and 10.

In one possible implementation, step S500 may include: the target image is divided into a plurality of pixel blocks. The difference value between the feature vectors corresponding to the pixel features of different pixel points in each pixel block is smaller than a preset difference value. The pixel blocks may also be represented as superpixels in the related art, and may be segmented by a superpixel segmentation algorithm (for example, SLIC algorithm, simple linear iterativeclustering algorithm, simple linear iterative clustering algorithm) in the related art, which is not described herein in detail. And then, according to the second corresponding relation and the screened and amplified matching point set, adjusting the warping degree corresponding to each pixel block. For example, the warp degree of each pixel block may be estimated according to the second correspondence and the position of each pixel block in the image, and the embodiments of the present disclosure are not limited herein. For example, the above-mentioned segmentation reference for super-pixel segmentation may refer to whether the difference between feature vectors corresponding to the pixel features of each pixel point is smaller than a preset difference, where the specific value of the preset difference is not limited herein, and a developer may set the segmentation reference according to practical situations.

In a possible implementation manner, the adjusting the warpage level corresponding to each pixel block according to the second correspondence and the filtered and amplified matching point set may include:

and determining a third corresponding relation corresponding to each pixel block according to the screened and amplified image characteristic points in the matching point set. Wherein the third corresponding relation is used for representing the scaling, rotation and offset relation between the target image and the reference imageAt least one of (a) and (b). The third correspondence S may be expressed as:

wherein a represents a scale, b represents an angle of rotation, t _x 、t _y The offset distances of the x-axis (horizontal or vertical, depending on the developer's settings) and the y-axis (vertical or horizontal) in the image are shown, respectively. The S matrix may be used to represent the relationship of image scaling, rotation, and offset, so that the scaling, rotation, and offset of the target image may be adjusted based on the reference image. S can be solved by referring to the solving mode of the first corresponding relation, which can be expressed as +.>

Wherein B is a matrix similar to the solution of a above, embodiments of the disclosure are not described herein. And then determining the corresponding rotation angle of the target image according to the second corresponding relation. The target image may be rotated by θ degrees, +. >

Wherein h is ₇ 、h ₈ Respectively the first element and the second element of the third row in the second corresponding relation matrix. And then rotating the target image according to the rotation angle. After the rotation, a new horizontal axis of the target image is taken as a u-axis (or hereinafter, the rotated horizontal axis). And then determining the maximum projection distance and the minimum projection distance of the center points of the blocks corresponding to the pixel blocks on the horizontal axis after the target image rotates. Illustratively, since the farther pixel blocks along the u-axis are typically distorted to a higher degree, if they are not weighted, the realism of the image is easily reduced. In view of this, embodiments of the present disclosure will->

And->

The maximum value (which may be expressed as a maximum distance) and the minimum value (which may be expressed as a minimum distance) of the projection of the cluster center of the pixel block of the target image in the u-axis direction are expressed, respectively. And determining a third corresponding relation weight and a second corresponding relation weight corresponding to each pixel block according to the maximum projection distance and the minimum projection distance. />

μ _h (k)＝1-μ _s (k) Wherein μ is _h Mu, as the second correspondence weight above _s The third correspondence weight above, the sum of which is 1./ >

Representing the pixel point corresponding to the minimum projection distance,

Pixel point corresponding to maximum projection distance, < >>

And the cluster center corresponding to the kth pixel block. />

And representing the length of the vector constructed according to the pixel point corresponding to the minimum projection distance and the clustering center corresponding to the kth pixel block.

The length of the vector constructed according to the pixel point corresponding to the minimum projection distance and the pixel point corresponding to the maximum projection distance is represented.<·>Representing an inner product operation. And finally, determining the warping degree corresponding to each pixel block according to the second corresponding relation, the second corresponding relation weight, the third corresponding relation and the third corresponding relation weight corresponding to each pixel block. Illustratively, the kth pixel block corresponds to a degree of warp +>

Can be expressed as +.>

Wherein mu _h Mu, as the second correspondence weight above _s The third correspondence weight above, the sum of which is 1./>

Is a second correspondence. Accordingly, the warping degree of each pixel block in the target image can be determined, and the image alignment between the target image and the reference image is realized.

In a possible implementation manner, the adjusting the warpage level corresponding to each pixel block according to the second correspondence and the filtered and amplified matching point set further includes: the reference image is divided into a plurality of pixel blocks. For example, the reference image may be segmented by an image segmentation algorithm used in the stage of dividing the target image into a plurality of pixel tiles. And then, according to the warping degree corresponding to each pixel block in the target image, adjusting the warping degree corresponding to each pixel block in the reference image. Illustratively, if the degree of warping of the target image is w (above

Can be represented as a matrix), the degree of warping of the reference image can be w.H- ¹ (H is a second correspondence).

It should be understood that, in some image stitching examples, the reference image may not be adjusted in terms of the warp degree, and the adjustment of the warp degree of the reference image is not limited to better stitching effect, and a developer may set the reference image according to an actual situation or identify the image to be stitched based on a specific algorithm, and then automatically determine whether to adjust the warp degree of the reference image according to the specific algorithm.

With continued reference to fig. 1, in step S600, the adjusted at least two images to be stitched are stitched to obtain a composite image. For example, the image to be stitched may be processed by any algorithm that can enhance the image stitching effect in the related art, such as performing eclosion fusion or the like on the image to be stitched.

And in combination with an actual application scene, the texture enhancement mode of the fractional differential texture enhancement can be used for enhancing the texture of the image to be spliced, and then the image warping is guided by a super-pixel division mode. The image processing method provided by the embodiment of the disclosure has the following advantages: 1. and a fractional differential texture enhancement mode is used for improving the step-by-step image texture in the process of constructing the image pyramid, so that the number of feature points which can be extracted is obviously increased. Especially in a low-texture scene, when a traditional method cannot extract enough feature points, the embodiment of the disclosure can increase the number of feature point extraction. 2. And adopting a fractal amplification mode in similar geometry to gradually amplify the number of matching points in the image in a fractional manner. In a scene with uneven textures, the distribution of matching points is often uneven, the image alignment effect is good at the places where the matching points are dense, and the object alignment effect is poor at the places where the matching points are sparse. The fractal amplification strategy can enable the distribution of matching points between images to be more uniform, and has better effect in subsequent alignment. 3. The image warping is guided by utilizing super-pixel division, and the local homography and local similar warping models can realize accurate pixel alignment in an image overlapping area and can keep a good extrapolation relation in a non-overlapping area. The image is divided into super-pixel guide warping similar to cell arrangement, so that the cost of mapping time can be saved, and distortion holes can be prevented from occurring along the outline of the object.

It will be appreciated that the above-mentioned method embodiments of the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic, and are limited to the description of the present disclosure. It will be appreciated by those skilled in the art that in the above-described methods of the embodiments, the particular order of execution of the steps should be determined by their function and possible inherent logic.

In addition, the disclosure further provides an image processing device, an electronic device, a computer readable storage medium, and a program, where the foregoing may be used to implement any one of the image processing methods provided in the disclosure, and the corresponding technical schemes and descriptions and corresponding descriptions referring to the method parts are not repeated.

Referring to fig. 11, fig. 11 shows a block diagram of an image processing apparatus provided according to an embodiment of the present disclosure, and as shown in fig. 11, the processing apparatus 100 includes: an image acquisition module 110, configured to acquire at least two images to be stitched; the feature point determining module 120 is configured to determine an image feature point corresponding to each image to be stitched; the matching point set construction module 130 is configured to perform similarity matching on image feature points in different images to be spliced, and construct a matching point set according to image feature points successfully matched between the different images to be spliced; the matching point set amplifying module 140 is configured to amplify the image feature points of the matching point set; the warping degree adjusting module 150 is configured to adjust the warping degree of at least one image to be spliced according to the amplified matching point set; the composite image generating module 160 is configured to perform image stitching on the at least two adjusted images to be stitched to obtain a composite image.

In a possible implementation manner, after the image feature point amplification is performed on the matching point set, the processing device further includes: the screening module is used for establishing a second corresponding relation according to the amplified multiple groups of image characteristic points in the matching point set; determining the mapping error of each group of image characteristic points in the amplified matching point set according to the second corresponding relation; screening the image characteristic points in the amplified matching point set according to the mapping error; the step of adjusting the warping degree of at least one image to be spliced according to the amplified matching point set comprises the following steps: and adjusting the warping degree of at least one image to be spliced according to the screened and amplified matching point set.

The method has specific technical relevance with the internal structure of the computer system, and can solve the technical problems of improving the hardware operation efficiency or the execution effect (including reducing the data storage amount, reducing the data transmission amount, improving the hardware processing speed and the like), thereby obtaining the technical effect of improving the internal performance of the computer system conforming to the natural law.

In some embodiments, functions or modules included in an apparatus provided by the embodiments of the present disclosure may be used to perform a method described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.

The disclosed embodiments also provide a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method. The computer readable storage medium may be a volatile or nonvolatile computer readable storage medium.

The embodiment of the disclosure also provides an electronic device, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

Embodiments of the present disclosure also provide a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, performs the above method.

The electronic device may be provided as a server or other modality of device.

Fig. 12 shows a block diagram of an electronic device 1900 provided in accordance with an embodiment of the disclosure. For example, electronic device 1900 may be provided as a server or terminal device. Referring to fig. 12, electronic device 1900 includes a processing component 1922 that further includes one or more processors and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by processing component 1922. The application programs stored in memory 1932 may include one or more modules each corresponding to a set of instructions. Further, processing component 1922 is configured to execute instructions to perform the methods described above.

The electronic device 1900 may also include a power supply assembly 1926 configuredTo perform power management of the electronic device 1900, a wired or wireless network interface 1950 is configured to connect the electronic device 1900 to a network, and an input/output interface 1958. Electronic device 1900 may operate an operating system based on memory 1932, such as the Microsoft Server operating system (Windows Server) ^TM ) Apple Inc. developed graphical user interface based operating System (Mac OS X ^TM ) Multi-user multi-process computer operating system (Unix) ^TM ) Unix-like operating system (Linux) of free and open source code ^TM ) Unix-like operating system (FreeBSD) with open source code ^TM ) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 1932, including computer program instructions executable by processing component 1922 of electronic device 1900 to perform the methods described above.

The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

Computer program instructions for performing the operations of the present disclosure can be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.

The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

If the technical scheme of the application relates to personal information, the product applying the technical scheme of the application clearly informs the personal information processing rule before processing the personal information, and obtains independent consent of the individual. If the technical scheme of the application relates to sensitive personal information, the product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'explicit consent'. For example, a clear and remarkable mark is set at a personal information acquisition device such as a camera to inform that the personal information acquisition range is entered, personal information is acquired, and if the personal voluntarily enters the acquisition range, the personal information is considered as consent to be acquired; or on the device for processing the personal information, under the condition that obvious identification/information is utilized to inform the personal information processing rule, personal authorization is obtained by popup information or a person is requested to upload personal information and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing mode, and a type of personal information to be processed.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method of processing an image, the method comprising:

acquiring at least two images to be spliced;

determining image characteristic points corresponding to each image to be spliced;

performing similarity matching on image feature points in different images to be spliced, and constructing a matching point set according to the successfully matched image feature points among the different images to be spliced;

amplifying the image characteristic points of the matching point set;

according to the amplified matching point set, adjusting the warping degree of at least one image to be spliced;

and performing image stitching on the at least two adjusted images to be stitched to obtain a composite image.

2. The processing method of claim 1, wherein said performing image feature point amplification on said set of matching points comprises:

determining a reference image and a target image in the at least two images to be spliced, and determining a first corresponding relation of image feature points between the reference image and the target image according to the matching point set; wherein the target image is subjected to image adjustment by taking a reference image as a datum; the first corresponding relation is established based on a plurality of groups of image characteristic points in the matching point set;

and amplifying the image characteristic points of the matching point set according to the first corresponding relation, the pixel points in the reference image and the pixel points in the target image.

3. The processing method according to claim 2, wherein the performing image feature point amplification on the matching point set according to the first correspondence, the pixel point in the reference image, and the pixel point in the target image includes:

the following steps are executed until a preset condition is reached: dividing the target image into a plurality of target sub-images; taking the pixel point with the minimum pixel gradient in each target sub-image as the target pixel point of each target sub-image; the pixel gradient is used for representing the variation degree of pixel values among pixel points; determining a corresponding reference pixel point of each target pixel point in a reference image according to the first corresponding relation; taking the pixel point which is in the preset range of each reference pixel point, has the highest target pixel point similarity corresponding to each reference pixel point and is higher than a preset similarity threshold value as a new reference pixel point corresponding to each target pixel point, and adding each new reference pixel point and the target pixel point corresponding to each new reference pixel point as a group of successfully matched image feature points into the matching point set; and aiming at the target sub-image corresponding to each new reference pixel point, taking the target sub-image as a new target image, and re-executing the division of the target image into a plurality of target sub-images.

4. A processing method according to claim 3, wherein said determining, according to the first correspondence, a corresponding reference pixel point of each of the target pixel points in the reference image includes:

determining a weight matrix corresponding to each target pixel point according to the distance between the target pixel point and each image characteristic point in the target image;

and determining a corresponding reference pixel point of each target pixel point in the reference image according to the weight matrix, the first corresponding relation and the position of each target pixel point in the target image.

5. The processing method according to any one of claims 2 to 4, characterized in that after said subjecting the matching point set to image feature point amplification, the processing method further comprises:

establishing a second corresponding relation according to the amplified multiple groups of image characteristic points in the matching point set;

determining the mapping error of each group of image characteristic points in the amplified matching point set according to the second corresponding relation;

screening the image characteristic points in the amplified matching point set according to the mapping error;

the step of adjusting the warping degree of at least one image to be spliced according to the amplified matching point set comprises the following steps:

And adjusting the warping degree of at least one image to be spliced according to the screened and amplified matching point set.

6. The processing method according to claim 5, wherein said adjusting the warping degree of at least one of the images to be stitched based on the screened, amplified set of matching points comprises:

dividing the target image into a plurality of pixel blocks; the difference value between the feature vectors corresponding to the pixel features of different pixel points in each pixel block is smaller than a preset difference value;

and adjusting the warping degree corresponding to each pixel block according to the second corresponding relation and the screened and amplified matching point set.

7. The processing method of claim 6, wherein adjusting the warpage level corresponding to each pixel block according to the second correspondence and the filtered amplified matching point set comprises:

determining a third corresponding relation corresponding to each pixel block according to the screened and amplified image characteristic points in the matching point set; wherein the third correspondence is used to represent at least one of scaling, rotation, offset between the target image and the reference image;

Determining a rotation angle corresponding to the target image according to the second corresponding relation;

rotating the target image according to the rotation angle;

determining the maximum projection distance and the minimum projection distance of a plurality of block center points corresponding to the pixel blocks on a horizontal axis after the target image rotates;

determining a third corresponding relation weight and a second corresponding relation weight corresponding to each pixel block according to the maximum projection distance and the minimum projection distance;

and determining the warping degree corresponding to each pixel block according to the second corresponding relation, the second corresponding relation weight, the third corresponding relation and the third corresponding relation weight corresponding to each pixel block.

8. The method of claim 7, wherein adjusting the degree of warping corresponding to each pixel block according to the second correspondence and the filtered and amplified set of matching points further comprises:

dividing the reference image into a plurality of pixel blocks;

and adjusting the warping degree corresponding to each pixel block in the reference image according to the warping degree corresponding to each pixel block in the target image.

9. The processing method according to any one of claims 1 to 8, wherein the determining the image feature point corresponding to each image to be stitched includes:

generating a first image corresponding to each image to be spliced in each scale space which is sequentially arranged aiming at each image to be spliced; the scale space is used for adjusting the image size of the images to be spliced; the arrangement sequence of the scale space is inversely related to the image reduction coefficient corresponding to the scale space;

for each first image, generating a plurality of second images with different scale parameters in the same scale space according to a plurality of scale parameters which are sequentially arranged; the scale parameter is used for adjusting the resolution of the first image; the arrangement sequence of the scale parameters is inversely related to the corresponding fuzzy degree of the scale parameters;

enhancing the image texture of each second image;

aiming at each image to be spliced, generating a differential image corresponding to each image to be spliced in each scale space according to the adjacent enhanced second images of each image to be spliced in the same scale space;

and screening pixel points in the differential images, and taking the screened pixel points as image characteristic points corresponding to each image to be spliced.

10. The processing method of claim 9, wherein said filtering pixels in said differential image comprises:

comparing each pixel point with adjacent pixel points in the differential image, first pixel points at corresponding positions in the adjacent scale space and adjacent pixel points in the adjacent scale space, and determining extreme points in each pixel point; wherein the extreme point is a minimum pixel value point or a maximum pixel value point;

and taking the extreme point as the pixel point after screening under the condition that the position corresponding to the extreme point is positioned in a non-edge area in the differential image or the pixel value corresponding to the extreme point is higher than a preset value.

11. An image processing apparatus, characterized in that the processing apparatus comprises:

the image acquisition module is used for acquiring at least two images to be spliced;

the characteristic point determining module is used for determining image characteristic points corresponding to each image to be spliced;

the matching point set construction module is used for carrying out similarity matching on the image characteristic points in different images to be spliced and constructing a matching point set according to the successfully matched image characteristic points among the different images to be spliced;

The matching point set amplifying module is used for amplifying the image characteristic points of the matching point set;

the warping degree adjusting module is used for adjusting the warping degree of at least one image to be spliced according to the amplified matching point set;

and the synthetic image generation module is used for carrying out image stitching on the at least two adjusted images to be stitched to obtain a synthetic image.

12. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the instructions stored in the memory to perform the method of processing an image according to any of claims 1 to 10.

13. A computer-readable storage medium, on which computer program instructions are stored, characterized in that the computer program instructions, when executed by a processor, implement the method of processing an image according to any one of claims 1 to 10.