WO2023206475A1

WO2023206475A1 - Image processing method and apparatus, electronic device and storage medium

Info

Publication number: WO2023206475A1
Application number: PCT/CN2022/090570
Authority: WO
Inventors: 刘阳晨旭; 石啟凡; 江浩; 张锐; 王宇; 许兴涛
Original assignee: 北京小米移动软件有限公司
Priority date: 2022-04-29
Filing date: 2022-04-29
Publication date: 2023-11-02

Abstract

Embodiments of the present disclosure provide an image processing method and apparatus, an electronic device and a storage medium. The method comprises: obtaining a first image and a second image that comprise a same target object; adjusting the position of a second pixel in the target object in the second image to the same position as a first pixel in the target object in the first image in a first direction to obtain a third image, the first pixel and the second pixel being pixels at a same position in the target object; determining a target grid in the third image according to a reference grid in the first image, the target grid comprising a target position, and the target pixel value of a pixel corresponding to the target position and the reference pixel value of a pixel corresponding to a reference position in the reference grid meet a preset condition; and adjusting the position of the second pixel in the third image to the same position as the first pixel in the first image in a second direction according to the positional correspondence between the reference grid and the target grid. By means of the method, the alignment effect of the target object in the first image and the second image is improved.

Description

Image processing methods, devices, electronic equipment and storage media

Technical field

The present disclosure relates to the field of image processing but is not limited to the field of image processing, and in particular, to an image processing method, device, electronic equipment and storage medium.

Background technique

As the application of image processing technology becomes more and more widespread, many devices have image acquisition components. After the image acquisition component collects the image, it needs to process the image to obtain the image that meets the needs. The images that different image acquisition components can obtain when collecting images may be different. For example, in the same shooting scene, images of the same target object are collected, different image acquisition modules or the same image acquisition module uses different configuration parameters. Under this condition, the collected image of the target object will also be different.

Contents of the invention

The present disclosure provides an image processing method, device, electronic equipment and storage medium.

A first aspect of an embodiment of the present disclosure provides an image processing method, including: acquiring a first image and a second image; wherein both the first image and the second image include the same target object; in the first direction, adjust the position of the second pixel in the target object in the second image to the same position as the first pixel in the target object in the first image, to obtain a third image; wherein, The first pixel and the second pixel are pixels at the same position in the target object; the target grid in the third image is determined according to the reference grid in the first image; wherein, The target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and a reference pixel value of a pixel corresponding to the reference position in the reference grid that satisfies a preset condition; according to the reference grid and the The position correspondence relationship of the target grid is to adjust the position of the second pixel in the third image to the same position as the first pixel in the first image in the second direction; wherein, the The second direction is perpendicular to the first direction.

In some embodiments, determining the target grid in the third image based on the reference grid in the first image includes: creating M*N preset sizes in the first image a reference grid; according to the reference pixel values of the K pixels corresponding to the reference positions in the reference grid, determine in the third image that the K reference pixel values satisfy the preset condition K target pixel values; according to the K target pixel values, determine K target pixels corresponding to the target pixel value in the third image; determine the target network according to the K target pixels Grid; M, N and K are all positive integers.

In some embodiments, the method further includes: determining the first average pixel value of each pixel in the preset area; wherein the preset area includes: centered on the pixel corresponding to the reference position, the preset number of pixels is an area of radius; determining the first average pixel value as the reference pixel value.

In some embodiments, based on the reference pixel values of the pixels corresponding to the K reference positions in the reference grid, K target pixel values that meet the preset conditions with the K reference pixel values are determined in the third image, The method includes: determining K candidate pixel values in the third image; determining a second average pixel value of each pixel in the candidate area; wherein the candidate area includes: corresponding to the candidate pixel value The pixel is the center, and the preset number of pixels is the area of radius; the candidate pixel values corresponding to the K second average pixel values whose K reference pixel values meet the preset condition are, Determine the target pixel value.

In some embodiments, the reference grid at least includes: a square grid or a triangular grid; the reference position at least includes: a corner point of the grid or a midpoint of an edge of the grid.

In some embodiments, the preset condition at least includes: the sum of absolute values of differences between the reference pixel value and the target pixel value is less than a target threshold.

In some embodiments, according to the position corresponding relationship between the reference grid and the target grid, the position of the second pixel in the third image is adjusted to be consistent with the first pixel in the second direction. The position in the first image is the same, including: determining the position correspondence relationship according to the reference position and the target position; the position correspondence relationship includes: a mapping transformation relationship in the second direction; according to the second The mapping transformation relationship in the direction, in the second direction, adjusts the position of the second pixel located in the target grid in the third image to be consistent with that of all the pixels located in the reference grid. The positions of the first pixels in the first image are the same.

In some embodiments, the mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.

In some embodiments, in the first direction, the position of the second pixel in the target object in the second image is adjusted to be consistent with the position of the first pixel in the target object in the first image. have the same position, and obtain the third image, including: extracting the first feature point of the target object in the first image; extracting the second feature point of the target object in the second image; wherein, The first feature point and the second feature point are matching feature point pairs at the same position in the target object; the mapping transformation relationship in the first direction is determined based on the first feature point and the second feature point. ;According to the mapping transformation relationship in the first direction, in the first direction, adjust the position of the second pixel in the target object in the second image to be consistent with the position of the first pixel in the target object The third image is obtained at the same position in the first image.

In some embodiments, the mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.

A second aspect of an embodiment of the present disclosure provides an image processing device, including: an image acquisition module, configured to acquire a first image and a second image; wherein the first image and the second image both include the same target. Object; a first processing module configured to adjust the position of the second pixel in the target object in the second image in the first direction to the position of the first pixel in the target object in the first image. The positions in are the same, and a third image is obtained; wherein the first pixel and the second pixel are pixels at the same position in the target object; a target grid determination module is used to determine according to the first image Reference grid, determine the target grid in the third image; wherein, the target grid includes: target position; target pixel value of the pixel corresponding to the target position, and the reference position in the reference grid The reference pixel value of the corresponding pixel satisfies the preset condition; the second processing module is configured to convert the second pixel in the second direction according to the position correspondence relationship between the reference grid and the target grid. The position in the third image is adjusted to be the same as the position of the first pixel in the first image; wherein the second direction is perpendicular to the first direction.

A third aspect of the embodiment of the present disclosure provides an electronic device, including:

A processor and a memory used to store executable instructions that can be run on the processor, wherein: when the processor is used to run the executable instructions, the executable instructions execute the method described in any of the above embodiments.

A fourth aspect of the embodiments of the present disclosure provides a non-transitory computer-readable storage medium. Computer-executable instructions are stored in the computer-readable storage medium. When the computer-executable instructions are executed by a processor, any of the above are implemented. methods described in the examples.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

The technical solution of the embodiment of the present disclosure is to adjust the position of the second pixel of the target object in the second image to be the same as the position of the first pixel in the first image in one direction. Since the first image and the second image are planar images, the positions of the pixels can be determined in two directions, and in the other direction, the adjusted second image can be adjusted again according to the positional relationship between the preset grid and the target grid. Adjustment, since the grid includes multiple pixels, this can reduce the determination of the position correspondence of a single pixel in the other direction, thereby reducing the amount of calculations required to adjust the pixel position of the target object in the other direction.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the embodiments of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the embodiments of the invention.

Figure 1 is a schematic flowchart of an image processing method according to an exemplary embodiment;

Figure 2 is a schematic diagram of a first image and a second image according to an exemplary embodiment;

Figure 3 is a schematic diagram of adjusting the position of the second pixel in the first direction according to an exemplary embodiment;

Part (a) of Figure 4 is a schematic diagram of a first image according to an exemplary embodiment;

Part (b) of Figure 4 is a schematic diagram of a second image according to an exemplary embodiment;

Figure 5 is a schematic diagram after adjustment in the first direction according to an exemplary embodiment;

Figure 6 is a schematic diagram of determining a target grid according to an exemplary embodiment;

Part (a) of Figure 7 is a schematic diagram of a first image including a reference grid according to an exemplary embodiment;

Part (b) of Figure 7 is a schematic diagram of a second image including a target grid according to an exemplary embodiment;

Figure 8 is a schematic diagram of another image processing method according to an exemplary embodiment;

Figure 9 is a schematic diagram of determining a target pixel value according to an exemplary embodiment;

Figure 10 is a schematic diagram of adjustment in the second direction according to an exemplary embodiment;

Figure 11 is a schematic diagram of an image processing device according to an exemplary embodiment;

Figure 12 is a schematic diagram of another image processing method according to an exemplary embodiment;

Figure 13 is a block diagram of a terminal device according to an exemplary embodiment.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the invention. Rather, they are merely examples of apparatus and methods consistent with some aspects of embodiments of the invention.

The terminology used in the embodiments of the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of the present disclosure. As used in this disclosure, the singular forms "a," "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be called second information, and similarly, the second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "when" or "in response to determining."

In image processing, such as high dynamic range imaging (High Dynamic Range, HDR), multi-frame image super-resolution and image fusion, it is necessary to use the image information of multiple images to fuse to obtain an image with improved image quality. In this process Multiple images need to be aligned first to achieve alignment between image pixels, and then subsequent processing can be carried out.

Normally, alignment between image pixels can be achieved through dense alignment methods. Most of these methods are based on dense optical flow methods for image alignment. This method requires determining the correspondence between pixels in a two-dimensional direction. Due to performance limitations, most of them can only handle images with smaller resolutions; at the same time, this method requires that the initial misaligned pixel distance is within a certain range, so when the pixel differences in multiple images are large, such as with large amplitude When the field of view changes, the alignment effect is poor.

It can also be achieved through the grid alignment method, which is mostly used in the field of image splicing. By using the correspondence between feature points in the image as a condition to constrain the grid, the grid is determined based on the characteristic feature points and the movement of the grid points is constrained. Alignment of images. This method requires the distribution of feature points across the entire image, so when aligning images with fewer feature points, the alignment effect is poor. For example, in images with weak textures where it is difficult to extract image feature points, it is difficult to obtain better image alignment effects.

Referring to Figure 1, a schematic flow chart of an image processing method is provided as an example of the present disclosure. The method includes:

Step S100, obtain a first image and a second image; wherein both the first image and the second image include the same target object.

Step S200, in the first direction, adjust the position of the second pixel in the target object in the second image to the same position as the first pixel in the target object in the first image, to obtain a third image; wherein, the first The pixel and the second pixel are pixels at the same position in the target object.

Step S300, determine the target grid in the third image based on the reference grid in the first image; wherein the target grid includes: the target position; the target pixel value of the pixel corresponding to the target position, and the target pixel value in the reference grid. The reference pixel value of the pixel corresponding to the position meets the preset conditions.

Step S400, according to the positional relationship between the reference grid and the target grid, adjust the position of the second pixel located in the target grid in the third image in the second direction to be consistent with the position of the first pixel located in the reference grid. The position in the first image is the same; where the second direction is perpendicular to the first direction.

The method of this disclosed example can be applied at least in user terminals, that is, the execution subject of the method can at least include user terminals, and user terminals can include mobile terminals and fixed terminals. Mobile terminals can include mobile phones, tablet computers, vehicle-mounted central control devices, wearable devices, smart devices, etc. Smart devices can also include smart office equipment and smart home equipment.

The first image and the second image acquired in step S100 may be two-dimensional plane images, and both the first image and the second image include at least one identical target object. That is, the first image and the second image include common image information and are not exactly the same two images. The target object can be determined based on the actual image collection scene. For example, it can be the object of the captured image in any scene such as people, animals, plants, mountains, and rivers.

The first image and the second image may be images obtained under different image acquisition parameters. They may be images acquired by the same image acquisition module under different image acquisition parameters, including images of the same target object. The image acquisition parameters may be the field of view. angle and focal length etc.

The first image and the second image may also be images of the same target object collected by different image acquisition modules. For example, the first image acquisition module and the second image acquisition module respectively capture the same scene under different image acquisition parameters. The captured image contains the target object in the scene.

The image acquisition parameters can be determined according to the configuration parameters of the image acquisition module and the image acquisition angle. The first image and the second image only need to include the same part of image information, that is, there is common image information between the first image and the second image, and this part of the common image information is the target object.

For example, if the first image includes person A and tree B, and the second image includes person A and person C, then the target object is person A. The presentation effects of person A in the first image and person A in the second image may be different. For example, the magnification factor of person A in the first image is greater than the magnification factor of person A in the second image, and the magnification factor of person A in the first image may be different. The position in the image is different from the position of person A in the second image.

Refer to Figure 2, which is a schematic diagram of a first image and a second image. Part (a) of Figure 2 is a schematic diagram of a target object. Part (b) of Figure 2 is a schematic diagram of the first image. Part (c) of Figure 2 Part is a schematic diagram of the second image.

The target objects are trees and people, the first image and the second image are images of the target object at different magnifications, and both the first image and the second image include the same target object. For example, the first image may be an image of the target object collected by an image acquisition module with a magnification of 1 times, and the second image may be an image of the target object collected by an image acquisition module with a magnification of 3 times. The first image is an image of the target object at 1x magnification (1X), and the second image is an image of the target object at 3x magnification (3X). The first image includes more image information than the second image. That is, in addition to the image information included in the second image, the first image also includes image information that is not included in the second image.

In this embodiment, the image information included in the second image is the image information common to the first image and the second image, and the content displayed in the second image is the target object. The target object includes trees and people, and includes other than Background image beyond trees and people.

For step S200, after obtaining the first image and the second image, since any pixel in the image has a corresponding position, the position of the pixel in the image can be determined through the positions in any two directions, for example, through the coordinates The system can determine the position of any pixel in the first image in the first image, and can also determine the position of any pixel in the second image in the second image. A coordinate system can be determined from two directions.

The first direction can be any direction, and the inclination angle between the first direction and the second direction is not limited, as long as the first direction is perpendicular to the second direction. For example, the first direction is a vertical direction, the second direction is a horizontal direction, or the second direction is a vertical direction, the first direction is a horizontal direction, etc. When the first direction is perpendicular to the second direction, after adjusting the position of the second pixel in the first direction, and then adjusting the position of the second pixel in the second direction again, the position of the second pixel in the first direction will not be changed. position, thereby making it easy to adjust the position of the second pixel in the second direction alone and improving the accuracy of adjusting the position of the second pixel.

The pixels of the target object in the first image are first pixels, and the pixels of the target object in the second image are second pixels. For a certain position in the target object, the pixel corresponding to the position in the first image is the first pixel, and the pixel corresponding to the position in the second image is the second pixel. That is, the first pixel and the second pixel are pixels at the same position in the target object. For a certain target position in the target object, the first pixel and the second pixel match, and both the first pixel and the second pixel are the target. The position in pixels.

For example, among the trees in the target object shown in Figure 2, the pixel of the tree tip in the first image is the first pixel, and the pixel in the second image is the second pixel.

In the first direction, the position of the second pixel in the target object in the second image is adjusted to the same position as the position of the first pixel in the target object in the first image, and the adjusted second image is used as the third image. In the adjusted second image, each pixel in the target object is aligned in the first direction with the corresponding pixel of the target object in the first image, and the aligned first pixel and the second pixel have the same position in the first direction. .

For example, when the target object is the tree shown in Figure 2, the first direction is the vertical direction, and the position in the vertical direction is represented by y. The position of the tree tip in the first direction of the corresponding pixel in the first image is y1, and the position of the tree tip in the first direction of the corresponding pixel in the second image is y2. In the first direction, y2 is adjusted to be the same as y1. Similarly, the position of each corresponding pixel of the target object in the first direction in the second image is adjusted to be the same as the position in the first direction in the first image. In this way, in the first direction, the position of the corresponding pixel of the target object in the second image is aligned with the position of the corresponding pixel in the first image, thereby aligning the position of the entire target object in the second image with the position of the corresponding pixel in the first image. Position alignment within an image.

For step S300, after the position of the second pixel in the first direction is aligned with the position of the first pixel, the target grid in the third image is determined based on the reference grid in the first image. The first image has a preset reference grid. The reference grid may also be called a reference window. The reference grid may be a virtual grid used to divide the first image into regions. There may be multiple reference grids in the first image, and the size and number of the reference grids may be determined according to actual usage requirements.

The shape of the reference grid can be a square grid or a triangular grid, etc., and can be determined according to actual needs. For example, there are N*N uniformly distributed square grids in the first image. The square grids may be positive direction grids, and the reference grid divides the first image into N*N grid areas.

The reference grid includes a reference position, which can be the corner point of the grid or the midpoint of the edge of the grid, etc., and can be adjusted according to actual needs. The reference position corresponds to a pixel in the first image, and the pixel value of the pixel corresponding to the reference position in the first image can be determined.

The target grid includes the target position, the target pixel value of the pixel corresponding to the target position, and the reference pixel value of the pixel corresponding to the reference position in the reference grid that meets the preset conditions. The target grid can be determined by the target position, and the preset condition can be determined according to actual needs. For example, the preset condition at least includes: the sum of the absolute values of the differences between the reference pixel value and the target pixel value is less than the target threshold.

Each reference grid corresponds to a target grid. This embodiment takes one of the grids as an example for explanation. The corresponding target grid can be determined based on each reference grid.

Step S400, after determining the target grid in the third image, adjust the position of the second pixel in the third image in the second direction to be consistent with the position of the reference grid and the target grid. A pixel has the same position in the first image.

The position correspondence between the reference grid and the target grid can be determined based on the reference position and the target position. The detailed determination process is not limited. You can refer to subsequent embodiments as long as the position correspondence can be determined.

For example, when the target object is the tree shown in Figure 2, the second direction is the horizontal direction, and the position in the horizontal direction is represented by x. The position of the tree tip in the first direction of the corresponding pixel in the first image is y1, and the position of the tree tip in the first direction of the corresponding pixel in the second image is y2. In the first direction, y2 is adjusted to be the same as y1. Similarly, the position of each corresponding pixel of the target object in the first direction in the second image is adjusted to be the same as the position in the first direction in the first image. In this way, in the first direction, the position of the corresponding pixel of the target object in the second image is aligned with the position of the corresponding pixel in the first image, thereby aligning the position of the entire target object in the second image with the position of the corresponding pixel in the first image. Position alignment within an image.

In this way, the position of the second pixel in both the first direction and the second direction is adjusted to be the same as the position of the first pixel in the first image, so that the position of each pixel in the target object in the second image is the same as the position in the second image. The positions in the first image are the same. When the positions are the same, the target object in the second image and the terminal target object in the first image are aligned.

On the one hand, first align the second pixel of the target object in the second image with the first pixel in the first image from the first direction, so that only the first pixel and the first pixel in the second direction are left. The two pixels are not aligned, so when determining the target grid in step S300, the pixel value of the pixel corresponding to the target position in the second direction and the pixel value of the pixel corresponding to the reference position can be separately considered, thereby increasing the computational complexity of determining the target grid. , efficiency and accuracy.

On the other hand, the target grid is determined according to the pixel value of the pixel corresponding to the reference position in the reference grid, and then the pixel position is adjusted in the second direction according to the position correspondence between the target grid and the reference grid, which reduces the need for The dependence on the feature points of the target object in the third image does not require determining the target grid based on the position of the feature points in the target object, nor does it need to be aligned in the second direction based on the position of the feature points of the target object in the third image. .

This reduces the situation where the accuracy of pixel position alignment in the target object in the second direction is poor when the feature points of the target object cannot be extracted in the third image or there are fewer feature points of the target object in the third image. When the feature points of the target object cannot be extracted in the third image or there are few feature points of the target object in the third image, the accuracy of pixel position alignment in the target object in the second direction is improved, thereby improving the alignment of the target object. Effect.

In another embodiment, refer to FIG. 3 , which is a schematic diagram of adjusting the position of the second pixel in the first direction. Step S200, in the first direction, adjust the position of the second pixel in the target object in the second image to the same position as the first pixel in the target object in the first image, to obtain a third image, including:

Step S201: Extract the first feature point of the target object in the first image.

Step S202, extract the second feature point of the target object in the second image; wherein the first feature point and the second feature point are matching feature point pairs at the same position in the target object. There is no necessary sequence relationship between step S201 and step S202. They can be executed at the same time, or any one of them can be executed first.

Step S203: Determine the mapping transformation relationship in the first direction based on the first feature point and the second feature point.

Step S204: According to the mapping transformation relationship in the first direction, adjust the position of the second pixel in the target object in the second image in the first direction to the same position as the position of the first pixel in the target object in the first image. , get the third image.

After obtaining the first image and the second image, extract the feature points of the target object in the first image and the feature points of the target object in the second image, and record the extracted feature points of the target object in the first image as the first feature point , record the extracted feature points of the target object in the second image as second feature points. The first feature point and the second feature point are matching feature point pairs corresponding to the feature at the same position in the target object. For the same feature in the target object, it is the first feature point in the first image and it is the first feature point in the second image. The second feature point, the first feature point and the second feature point are a pair of matching feature points.

Referring to Figure 4, part (a) of Figure 4 is a schematic diagram of a first image, and part (b) of Figure 4 is a schematic diagram of a second image. The house and the person are the target objects. The dots on the house and the dots on the person shown in the first image are the first features. The dots on the house and the dots on the person shown in the second image are as the second characteristic. The first feature and the second feature, which is a pair of feature points matching the first feature, are connected by a dotted line. For a certain first feature point of the target object in the first image, there is a second feature point matching the first feature point in the second image.

For example, when the feet of a character are used as feature points, the feature points of the feet in the first image are the first feature points, the feature points of the feet in the second image are the second feature points, and the dotted lines represent the features of the feet. The first feature point and the second feature point are connected, and the first feature point and the second feature point, which are both feature points of the foot, are a matching pair of feature points. The same applies to other feature points in Figure 4.

The feature points of the feet in the first image and the feature points of the character's hands in the second image are not matching feature point pairs. When the feature points of the feet in the first image are the first feature points, the character's hands in the second image The feature point of the part is not the second feature point that matches the first feature point.

There are multiple first feature points and multiple second feature points respectively, and the numbers of the first feature points and the second feature points may be the same. The method of extracting the first feature point and the second feature point can be determined according to the business needs, as long as the first feature point and the second feature point can be extracted, for example, the feature point extraction algorithm includes scale-invariant feature transformation (Scale-invariant feature transform, SIFT) algorithm and ORB (ORiented Brief) feature extraction algorithm, etc.

In step S203, the mapping transformation relationship in the first direction is determined based on the matching first feature points and second feature points.

After determining the first feature point and the second feature point, the position of the first feature point in the first image and the position of the second feature point in the second image can be obtained. According to multiple pairs of matching first features The position of the point and the second feature point can determine the mapping transformation relationship in the first direction.

For example, the mapping transformation relationship in the first direction can be determined based on the positions of four pairs of matching first feature points and second feature points, and the mapping transformation relationship can be a unitary transformation matrix. The homography transformation matrix can be a 3*3 matrix, which can be expressed by the following formula:

H ₁ represents the homography transformation matrix in the first direction; h ₀ to h ₈ represent the elements in H ₁ .

The mapping transformation relationship can also be other forms of transformation matrices. According to the matching positions of the first feature point and the position of the second feature point, the position of the second pixel in the second image can be adjusted in the first direction. That’s it.

Step S204: After obtaining the mapping transformation relationship in the first direction, adjust the position of the second pixel in the target object in the second image in the first direction to be consistent with the target object according to the mapping transformation relationship in the first direction. The position of the first pixel in the first image is the same, and the third image is obtained.

In another embodiment, the homography transformation matrix H ₁ in the first direction may also satisfy the following conditions:

i represents the matched i-th pair of feature points, that is, the i-th first feature point and the i-th second feature point that match, x _i represents the first feature point in the _i -th pair of feature points in the second direction Coordinates, y _i represents the coordinate of the first feature point in the i-th pair of feature points in the first direction, and y _i ' represents the coordinate of the second feature point in the i-th pair of feature points in the first direction.

When H ₁ meets this condition, according to H ₁ , the position of the second pixel in the target object in the second image can be adjusted in the first direction to the same position as the position of the first pixel in the target object in the first image, Get the third image.

In this embodiment, the first direction is a vertical direction.

When the first direction is the horizontal direction, the position in the first direction is represented by x, and the position in the second direction is represented by y. The homography transformation matrix H ₁ can also satisfy the following conditions:

x _i ' represents the coordinate of the second feature point in the i-th pair of feature points in the first direction.

Refer to Figure 5, which is a schematic diagram of alignment in the first direction. As shown in FIG. 5 , a schematic diagram is shown in which the position of each pixel of the target object in the first direction in the second image shown in part (b) of FIG. 4 is aligned with the position of the corresponding pixel in the first image. The first direction here is the vertical direction. In the first image and the second image, the positions of the pixels corresponding to the target object are on the same line, achieving line alignment.

In another embodiment, refer to FIG. 6 , which is a schematic diagram of determining a target grid. Step S300, determine the target grid in the third image based on the reference grid in the first image, including:

Step S301: Create M*N reference grids of preset sizes in the first image.

Step S302: Based on the reference pixel values of the pixels corresponding to the K reference positions in the reference grid, determine K target pixel values that meet the preset conditions with the K reference pixel values in the third image.

Step S303: Based on the K target pixel values, K target pixels corresponding to the target pixel values are determined in the third image.

Step S304, determine the target grid based on K target pixels; M, N and K are all positive integers. Through step S300, the target grid matching each reference grid can be determined.

For step S301, the reference grid in the first image may be created in advance or may be created when step S300 is performed. The size of the reference grid can be determined according to actual needs. For example, the size, shape, and quantity of the preset grid can be determined according to the preset configuration parameters. M and N can be equal.

The reference positions and the number of reference positions in the reference grid can also be determined according to actual needs. For example, when the reference grid is a square, the reference positions are the four corner points of the reference grid, that is, four reference positions are included.

Referring to FIG. 7 , part (a) of FIG. 7 is a first image including a reference grid, and part (b) of FIG. 7 is a second image including a target grid.

Part (a) of Figure 7 shows a reference grid. The positive direction grid including four dots around the character is one of the reference grids. The positions of the four dots are the reference grid. Reference location. For other reference grids, the four corner points of each reference grid are the reference locations. The number of reference grids is 5*5, and the first image is divided into 5*5 reference grids.

Step S302 is explained by taking one of the reference grids as an example. According to the reference pixel values of the pixels corresponding to the K reference positions in the reference grid, determine the K reference pixel values that meet the preset conditions in the third image. target pixel value.

After determining the reference grid and the reference position in the reference grid, the pixel corresponding to the reference position in the first image can be determined, and then the pixel value of the corresponding pixel in the first image for each reference position can be determined. Then, based on the preset conditions and the pixel values of the corresponding pixels in the K reference positions in the first image, K target pixel values that meet the preset conditions with the K reference pixel values in the third image can be determined.

When the preset conditions are different, the determined target pixel value may also be different, and the preset conditions can be determined according to actual needs.

For step S303, after the K target pixel values are determined, K target pixels corresponding to the K target pixel values can be determined in the third image based on the K target pixel values. One target pixel value can correspond to one target pixel.

For step S304, after K target pixels are determined, the target grid is determined based on the K target pixels; M, N, and K are all positive integers. The position of the target pixel in the third image is the target position in the target grid, and the target grid can be determined based on the target position. For example, one target pixel corresponds to one target position, K target pixels correspond to K target positions, and the area surrounded by the K target positions is determined as the target grid.

Referring to part (b) of Figure 7, the pixel value of the pixel corresponding to the four dots around the character is the target pixel value, the pixel corresponding to the four dots is the target pixel, and the positive direction surrounded by the target pixels is the target grid. .

For other reference grids, the process of determining the corresponding target grid is the same as the above process, that is, the above process applies to each reference grid.

In another embodiment, a target position corresponds to a reference position, and in the corresponding target position and reference position, the position of the target pixel and the position of the reference pixel are the same in the first direction.

Referring to Figure 7, in part (a) of Figure 7, the reference position corresponding to the upper left corner of the reference grid is recorded as the first reference position, and the reference position corresponding to the upper right corner is recorded as the second reference position. In part (b) of Figure 7 The position corresponding to the upper left corner of the target grid is recorded as the first target position, and the position corresponding to the upper right corner is recorded as the second target position. When the first direction is the vertical direction, the first target position and the first reference position are the same in the first direction, that is, they are row aligned and on the same row. The second target position and the second reference position are the same in the first direction, that is, row aligned, on the same row.

This can reduce the situation where the target position and the reference position deviate simultaneously in the first and second directions. After the positions in the first direction are the same, the position in the second direction is left with a one-dimensional variable, which can be convenient The position in the second direction is adjusted so that the positions of the target position and the reference position in the first direction and the second direction are adjusted to be the same.

In addition, it also reduces the problem of large calculation amount and low accuracy of position adjustment when performing step S400 when the corresponding target position and reference position are different in the first direction and the second direction. The amount of calculation is reduced and the accuracy of position adjustment is improved.

In another embodiment, referring to FIG. 8 , which is a schematic diagram of another image processing method, the method includes:

Step S10, determine the first average pixel value of each pixel in the preset area; wherein the preset area includes: an area with the pixel corresponding to the reference position as the center and the preset number of pixels as the radius;

Step S20, determine the first average pixel value as the reference pixel value.

When determining the reference pixel value, the reference pixel value is determined by combining the pixel value of the pixel corresponding to the reference position and the pixel values of the surrounding pixels corresponding to the reference position. The reference position may correspond to a certain pixel in the first image. With the pixel corresponding to the reference position as the center and the radius of the preset number of pixels as the radius, the pixel value of each pixel in the preset area is used as the basis. The average value of the pixels is determined as the reference pixel. Here, the average value of each pixel in the preset area is recorded as the first average pixel value.

This can easily improve the accuracy of determining the target position and reduce the problem of low accuracy caused by determining the target pixel value based on the pixel value of a single pixel corresponding to each reference position.

In another embodiment, referring to FIG. 9 , which is a schematic diagram of determining a target pixel value, step S302 includes:

Step S3021, determine K candidate pixel values in the third image.

Step S3022, determine the second average pixel value of each pixel in the candidate area; wherein the candidate area includes: an area with the pixel corresponding to the candidate pixel value as the center and a preset number of pixels as the radius.

Step S3023: Determine the candidate pixel values corresponding to the K second average pixel values whose K reference pixel values meet the preset conditions as target pixel values.

When determining the target pixel value, K candidate pixel values are first determined in the third image, and then the average pixel value of the pixel values of each pixel in the candidate area corresponding to each candidate pixel value is determined, that is, the second average Pixel values. One second average pixel value corresponds to one candidate pixel value, and the corresponding K candidate pixel values can be determined based on the K second average pixel values. Combining the preset conditions and the K reference pixel values, the candidate pixel values corresponding to the K second average pixel values whose K reference pixel values meet the preset conditions are determined as target pixel values.

This method improves the accuracy of determining the target pixel value and reduces the problem of low accuracy caused by determining the target pixel value based on the pixel value corresponding to a single pixel, thereby improving the accuracy of determining the target grid and further improving the accuracy of the target pixel value. In the second direction, the position of each pixel of the target object in the third image is adjusted to the same accuracy as the position of the corresponding pixel of the target object in the first image in the first image.

In another embodiment, refer to FIG. 10 , which is a schematic diagram of adjustment in the second direction. Step S400: According to the position correspondence relationship between the reference grid and the target grid, in the second direction, adjust the position of the second pixel located in the target grid in the third image to be consistent with the position of the first pixel located in the reference grid. The pixels are at the same position in the first image, including:

Step S401, determine the position correspondence relationship according to the reference position and the target position. The position correspondence relationship includes: the mapping transformation relationship in the second direction;

Step S402: According to the mapping transformation relationship in the second direction, adjust the position of the second pixel located in the target grid in the third image in the second direction to be in the same position as the first pixel located in the reference grid. The same position in an image.

For each reference grid and the target grid matching the reference grid, the position correspondence can be determined. A reference grid and a matching target grid form a grid pair. Through this method, the position correspondence relationship of each grid pair can be determined, that is, the mapping transformation relationship in the second direction.

In this embodiment, after the target grid is determined, the position correspondence relationship in the second direction can be determined based on the target position in the target grid and the reference position in the reference grid. The position in the second direction can be determined. The corresponding relationship includes: mapping transformation relationship in the second direction. The mapping transformation relationship in the second direction may be a transformation relationship in the form of a homography transformation matrix or the like.

Through the mapping transformation relationship in the second direction, the position of the second pixel of the target object in the target grid in the third image is adjusted, and the adjusted position of the second pixel of the target object in the target grid in the third image is, The second direction is the same as the position corresponding to the first pixel in the reference grid in the first image, so that the position of each pixel of the target object in the target grid is aligned with the position of the pixel corresponding to the target object in the reference grid. That is, according to the second upward mapping transformation relationship, the position of each pixel in the target grid is adjusted in the second direction.

Through the target grid corresponding to the M*N reference grids, the position of each pixel of the target object located in the target grid in the second direction can be compared with the corresponding pixel of the target object in the reference grid in the second direction. Align the position on.

For example, the target grid corresponding to the four dots where the character is located as shown in part (b) of Figure 7, and the position of each pixel in the target grid in the second direction are different from the target grid shown in part (a) of Figure 7. The corresponding pixels in the reference grid corresponding to the four dots where the character is located are aligned in the second direction. According to the mapping transformation relationship in the second direction, the position of the corresponding pixel in the second direction of the left fingertip of the person shown in part (b) of Figure 7 is compared with the position of the left fingertip of the person's left finger shown in part (a) of Figure 7 The positions of the corresponding pixels in the second direction are adjusted to be the same, that is, aligned. Align the position of the pixel corresponding to the fingertip of the person's right hand in the second direction shown in part (b) of Figure 7 with the position of the pixel corresponding to the fingertip of the person's right hand shown in part (a) of Figure 7 in the second direction. .

According to the mapping transformation relationship in the second direction, the position of the pixel corresponding to the person's head shown in part (b) of Figure 7 in the second direction is compared with the position of the pixel corresponding to the person's head shown in part (a) of Figure 7 in the second direction. Adjust the positions in the two directions to be the same, that is, align.

H ₂ represents the homography transformation matrix in the second direction; h ₀ to h ₈ represent the elements in H ₂ .

When the first direction is the vertical direction and the second direction is the horizontal direction, the homography transformation matrix H ₂ can also satisfy the following conditions:

When _H2 meets this condition, according to _H2 , the position of the second pixel located in the target grid in the third image in the second direction can be adjusted to be the same as the position of the first pixel located in the reference grid in the first image. are in the same position.

When the first direction is the horizontal direction, the position in the first direction is represented by x, and the position in the second direction is represented by y. The homography transformation matrix H ₂ can also satisfy the following conditions:

In another embodiment, referring to FIG. 11 , which is a schematic diagram of an image processing device, the device includes:

Image acquisition module 1, used to acquire a first image and a second image; wherein the first image and the second image include the same target object;

The first processing module 2 is configured to adjust the position of the second pixel in the target object in the second image in the first direction to the position of the first pixel in the target object in the first image. The positions are the same, and a third image is obtained; wherein the first pixel and the second pixel are pixels at the same position in the target object;

Target grid determination module 3, configured to determine the target grid in the third image according to the reference grid in the first image; wherein the target grid includes: target position; the target The target pixel value of the pixel corresponding to the position satisfies the preset conditions with the reference pixel values of the pixels corresponding to the N reference positions in the reference grid;

The second processing module 4 is configured to add the second pixels located in the target grid to the third image in the second direction according to the positional relationship between the reference grid and the target grid. The middle position is adjusted to the same position in the first image as the first pixel located within the reference grid; wherein the second direction is perpendicular to the first direction.

In another embodiment, the target grid determination module 3 includes:

A creation unit configured to create M*N reference grids of preset sizes in the first image;

A target pixel value determination unit configured to determine, in the third image, the K reference pixel values that satisfy the requirement based on the reference pixel values of the K pixels corresponding to the reference positions in the reference grid. K target pixel values of preset conditions;

A target pixel determination unit configured to determine K target pixels corresponding to the target pixel value in the third image based on the K target pixel values;

A target grid determining unit is used to determine the target grid based on K target pixels; M, N and K are all positive integers.

In another embodiment, the device further includes:

The first average pixel value determination module is used to determine the first average pixel value of each pixel in the preset area; wherein the preset area includes: with the pixel corresponding to the reference position as the center and the preset number of pixels as the radius. area;

A reference pixel value determination module, configured to determine the first average pixel value as the reference pixel value.

In another embodiment, the target pixel value determination unit includes:

Alternative pixel value determination subunit, used to determine K alternative pixel values in the third image;

The average pixel value determination subunit is used to determine the second average pixel value of each pixel in the candidate area; wherein the candidate area includes: centered on the pixel corresponding to the candidate pixel value, the preset The number of pixels is the area of the radius;

Target pixel value determination subunit, configured to determine the candidate pixel values corresponding to the K second average pixel values whose K reference pixel values satisfy the preset condition as the target pixels. value.

In another embodiment, the reference grid at least includes: a square grid or a triangular grid; the reference position at least includes: a corner point of the grid or a midpoint of an edge of the grid.

In another embodiment, the preset condition at least includes: the sum of absolute values of differences between the reference pixel value and the target pixel value is less than a target threshold.

In another embodiment, the second processing module 4 includes:

A first position correspondence determination unit, configured to determine the position correspondence according to the reference position and the target position; the position correspondence includes: a mapping transformation relationship in the second direction;

A first processing unit configured to position the second pixel located in the target grid in the third image in the second direction according to the mapping transformation relationship in the second direction, Adjusted to the same position in the first image as the first pixel located within the reference grid.

In another embodiment, the mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.

In another embodiment, the first processing module 2 includes:

A first feature point extraction unit configured to extract the first feature point of the target object in the first image;

A second feature point extraction unit is used to extract the second feature point of the target object in the second image; wherein the first feature point and the second feature point are at the same position in the target object. matching feature point pairs;

A second position correspondence relationship determination unit configured to determine the mapping transformation relationship in the first direction based on the first feature point and the second feature point;

A second processing unit configured to adjust the position of the second pixel in the target object in the second image in the first direction to be consistent with the position of the second pixel in the second image according to the mapping transformation relationship in the first direction. The position of the first pixel in the target object in the first image is the same, and the third image is obtained.

In another embodiment, the mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.

In another embodiment, an electronic device is also provided, including:

A processor and memory for storing executable instructions capable of running on said processor, wherein:

When the processor is used to run the executable instructions, the executable instructions execute the method described in any of the above embodiments.

In another embodiment, a non-transitory computer-readable storage medium is also provided. Computer-executable instructions are stored in the computer-readable storage medium. When the computer-executable instructions are executed by a processor, any of the above are implemented. methods described in the examples.

It should be noted that the “first” and “second” in the embodiments of the present disclosure are only for convenience of expression and distinction and have no other specific meanings.

In another embodiment, another image processing method is also provided. Refer to Figure 12, which is a schematic diagram of this method.

Under normal circumstances, global image alignment methods based on homography transformation are difficult to align objects on different planes; methods based on dense optical flow fields require a large amount of calculations and are difficult to process larger images; methods based on grid alignment are basically Feature points are used to constrain the displacement of the grid, and the alignment effect is poor in scenes where image feature points cannot be extracted.

The disclosed example can be applied to the field of image alignment, and can be used to align images of the same scene captured by a single camera at different viewing angles, or can be used to align images of the same scene captured by multiple cameras. The aligned images can be used as image input for multi-frame HDR and super-resolution algorithms, and can also be used as input for multi-camera image fusion algorithms.

Take the first image collected by the mobile phone camera at 1x magnification and the second image collected at 3x magnification as an example for explanation. As shown in Figure 4, when the 1x and 3x lenses of the mobile phone shoot the same scene, due to the difference in shooting angles and the differences in various configurations of the lenses, such as field of view, focal length, etc., the first image and the second image are in the same scene. There is a big difference in field of view size and other aspects. In order to use the images of the two lenses for image fusion, an algorithm needs to be used to align the first image and the second image before the subsequent algorithm processing process can be carried out.

Referring to Figure 12, the method includes:

Step 1: Extract feature points of the target object in the first image and the second image. The same feature of the target object forms a feature point pair in the first image and the second image.

Step 2: By matching the feature point pairs, determine that in the first direction, the position of the pixel of the target object in the second image and the position of the corresponding pixel of the target object in the first image are adjusted to the same homography transformation matrix, for example The first direction is the vertical direction, and the homography matrix of the global row alignment of the image is used to realize the row alignment of the target object in the first image and the target object in the second image, and obtain the third image.

Step 3: Determine the corresponding target grid in the third image based on the reference grid in the first image. It includes: dividing the grid on the third image, that is, the row-aligned image, and obtaining the target grid corresponding to the grid point on the image through row search and matching of grid points.

Step 4: For each reference grid and the corresponding target grid, calculate the position transformation relationship of the target object in the first image and the third image in the second direction using the corresponding relationship between the preset position in the grid and the target position. The position of the target object in the third image is adjusted according to the position transformation relationship.

Step 1: Extract matching feature points of the target object in the first image and the target object in the second image.

Since there may be differences in rotation, translation, field of view, etc. between the first image and the second image, in order to perform line-aligned global homography transformation on the first image and the second image, it is necessary to obtain the target object in the first image and the second image. Feature matching point pairs of target objects in two images. Common feature points and matching methods such as SIFT and ORB can be used. Refer to Figure 4.

Step 2: Calculate the global row-aligned homography matrix and perform row alignment on the image.

The global homography transformation matrix H, specifically a 3*3 matrix, has the following form:

h ₀ to h ₈ represent elements of H.

The homography matrix contains a total of 8 degrees of freedom, and a set of matching feature point pairs in step 1 can provide two sets of constraints. Therefore, at least four pairs of matching feature point pairs are required to obtain a row-aligned homography matrix. At the same time, in order to achieve row alignment of the image, the obtained H matrix also needs to satisfy the transformed image, and the coordinates (i.e. y coordinate) of the matching feature point pairs in the row direction are equal, and the H matrix satisfies:

That is, for the position (x, y) of the corresponding pixel of the feature point A shown in part (b) of Figure 4 and the position (x1, y1) of the corresponding pixel of the feature point A' shown in part (a) of Figure 4, After the position of the pixel corresponding to point A undergoes H transformation, y and y' are equal.

Step 3: On the third image after row alignment, use grid point row search to determine the target grid.

After using the global homography matrix H to achieve row alignment of the image, divide a uniform reference grid on the first image, and then use the sum of the absolute values of the differences in pixel values to search for the target grid on the row-aligned third image. , thereby obtaining the corresponding relationship between each grid point in the left and right images. Refer to Figure 7 for a schematic diagram of grid search matching.

Step 4: Use the reference position of the reference grid and the target position of the target grid to determine the position transformation relationship between the target object in the first image and the target object in the third image.

After step three, the corresponding relationship between the reference position in each reference grid and the target position in the target grid can be obtained. Since each grid has four grid points, homography transformation can be used to calculate each reference The position correspondence between the pixels in the grid and the corresponding target grid. The form of homography transformation is similar to the H matrix in step 2. By mapping the pixels in each target grid using homography transformation, the third image is aligned with the pixel positions of the target object in the first image.

1. Use global homography transformation for row alignment, limit image misalignment to the column direction, and limit the search range of grid points to one dimension. Compared with the grid alignment method based on optical flow, the alignment is reduced. amount of operations in the process.

2. Grid the image and directly search for grid matching points, which increases the reference information available for grid alignment, making the constraints between grid points independent of feature points based on grid alignment methods. The extraction situation improves the alignment effect.

Figure 13 is a block diagram of a terminal device according to an exemplary embodiment. For example, the terminal device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

Referring to Figure 13, the terminal device may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and communications component 816.

The processing component 802 generally controls the overall operations of the terminal device, such as operations associated with presentations, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

The memory 804 is configured to store various types of data to support operations at the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EEPROM), Programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Power component 806 provides power to various components of the terminal device. Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to end devices.

Multimedia component 808 includes a screen that provides an output interface between the terminal device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. A touch sensor can not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front-facing camera and rear-facing camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive external audio signals when the terminal device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in memory 804 or sent via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc. These buttons may include, but are not limited to: Home button, Volume buttons, Start button, and Lock button.

Sensor component 814 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor component 814 can detect the open/closed state of the terminal device, the relative positioning of components, such as the display and keypad of the terminal device, and the sensor component 814 can also detect the position change of the terminal device or a component of the terminal device, The presence or absence of user contact with the terminal device, terminal device orientation or acceleration/deceleration and temperature changes of the terminal device. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the terminal device and other devices. Terminal devices can access wireless networks based on communication standards, such as WiFi, 4G or 5G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communications component 816 also includes a near field communications (NFC) module to facilitate short-range communications. For example, the NFC module can be implemented based on frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the terminal device may be configured by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable Gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are implemented for executing the above method.

Other embodiments of the invention will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common common sense or customary technical means in the technical field that are not disclosed in the present disclosure. . It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It is to be understood that the present invention is not limited to the precise construction described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims

An image processing method, wherein the method includes:

Obtaining a first image and a second image; wherein both the first image and the second image include the same target object;

In the first direction, adjust the position of the second pixel in the target object in the second image to the same position as the first pixel in the target object in the first image to obtain a third image ; Wherein, the first pixel and the second pixel are pixels at the same position in the target object;

According to the reference grid in the first image, the target grid in the third image is determined; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and The reference pixel value of the pixel corresponding to the reference position in the reference grid meets the preset condition;

According to the position correspondence relationship between the reference grid and the target grid, in the second direction, the position of the second pixel in the third image is adjusted to be consistent with the position of the first pixel in the third image. The positions in an image are the same; wherein the second direction is perpendicular to the first direction.
The method of claim 1, wherein determining the target grid in the third image based on the reference grid in the first image includes:

Create M*N reference grids of preset sizes in the first image;

According to the reference pixel values of the K pixels corresponding to the reference positions in the reference grid, the K reference pixel values that satisfy the preset condition are determined in the third image. target pixel value;

According to the K target pixel values, K target pixels corresponding to the target pixel values are determined in the third image;

The target grid is determined based on K target pixels; M, N and K are all positive integers.
The method of claim 2, further comprising:

Determine the first average pixel value of each pixel in the preset area; wherein the preset area includes: an area with the pixel corresponding to the reference position as the center and the preset number of pixels as the radius;

The first average pixel value is determined as the reference pixel value.
The method according to claim 3, wherein, based on the reference pixel values of pixels corresponding to K reference positions in the reference grid, determining in the third image the values corresponding to the K reference positions. The K target pixel values whose pixel values meet the preset conditions include:

Determine K candidate pixel values in the third image;

Determine the second average pixel value of each pixel in the candidate area; wherein the candidate area includes: an area with the pixel corresponding to the candidate pixel value as the center and the preset number of pixels as the radius;

The candidate pixel values corresponding to the K second average pixel values whose K reference pixel values satisfy the preset condition are determined as the target pixel values.
The method according to any one of claims 1 to 4, wherein the reference grid at least includes: a square grid or a triangular grid;

The reference position at least includes: a corner point of the grid or a midpoint of an edge line of the grid.
The method according to any one of claims 1 to 4, wherein the preset condition at least includes: the sum of the absolute values of the differences between the reference pixel value and the target pixel value is less than a target threshold.
The method according to claim 1, wherein the position of the second pixel in the third image is adjusted in the second direction according to the position correspondence relationship between the reference grid and the target grid. to the same position as the first pixel in the first image, including:

The position correspondence relationship is determined according to the reference position and the target position; the position correspondence relationship includes: a mapping transformation relationship in the second direction;

According to the mapping transformation relationship in the second direction, in the second direction, the position of the second pixel located in the target grid in the third image is adjusted to be the same as that located in the reference The first pixels within the grid are at the same position in the first image.
The method according to claim 7, wherein the mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.
The method according to claim 1, wherein in the first direction, the position of the second pixel in the target object in the second image is adjusted to be consistent with the position of the first pixel in the target object. The same position in the first image is obtained, and the third image is obtained, including:

Extract the first feature point of the target object in the first image;

Extract the second feature point of the target object in the second image; wherein the first feature point and the second feature point are matching feature point pairs at the same position in the target object;

Determine a mapping transformation relationship in the first direction according to the first feature point and the second feature point;

According to the mapping transformation relationship in the first direction, in the first direction, the position of the second pixel in the target object in the second image is adjusted to be in the same position as the first pixel in the target object. The positions in the first image are the same, and the third image is obtained.
The method according to claim 9, wherein the mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.
An image processing device, wherein the device includes:

An image acquisition module, configured to acquire a first image and a second image; wherein both the first image and the second image include the same target object;

A first processing module configured to adjust the position of the second pixel in the target object in the second image in the first direction to the position of the first pixel in the target object in the first image. The positions are the same, and a third image is obtained; wherein the first pixel and the second pixel are pixels at the same position in the target object;

A target grid determination module, configured to determine the target grid in the third image according to the reference grid in the first image; wherein the target grid includes: a target position; the target position The target pixel value of the corresponding pixel satisfies the preset condition with the reference pixel value of the pixel corresponding to the reference position in the reference grid;

A second processing module configured to adjust the position of the second pixel in the third image in the second direction to be consistent with the first position according to the position correspondence between the reference grid and the target grid. The positions of the pixels in the first image are the same; wherein the second direction is perpendicular to the first direction.
An electronic device, including a processor, a transceiver, a memory, and an executable program stored on the memory and capable of being run by the processor, wherein when the processor runs the executable program, it executes claims 1 to 10 any of the methods provided.
A computer storage medium stores an executable program; after the executable program is executed by a processor, the method as provided in any one of claims 1 to 10 can be implemented.