CN115965675A

CN115965675A - Image processing method and device, electronic device and storage medium

Info

Publication number: CN115965675A
Application number: CN202111186783.3A
Authority: CN
Inventors: 张超
Original assignee: Beijing Xiaomi Mobile Software Co Ltd
Current assignee: Beijing Xiaomi Mobile Software Co Ltd
Priority date: 2021-10-12
Filing date: 2021-10-12
Publication date: 2023-04-14

Abstract

The disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium. The method comprises the following steps: acquiring two groups of combined images of an acquisition object; the method comprises the steps that one group of combined images comprise a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprise a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any group of combined images have a mapping relation; determining a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information characterizes a distance of the acquisition object from the first camera. By the method, the accuracy of homography estimation can be improved.

Description

Image processing method and device, electronic device and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

In the field of image processing, a homography matrix is defined as a projection mapping matrix of one plane to another. Assuming that a homography between the two images has been taken, the two images can be associated by means of a homography matrix H, for example by associating image a with image B by means of the formula a = H × B, i.e. by transforming image B into image a by means of a homography matrix H.

Homography matrices have many practical applications such as correction, registration of images, and estimation of motion between two cameras in SLAM. However, how to accurately estimate the homography matrix is critical.

Disclosure of Invention

The disclosure provides an image processing method and device, an electronic device and a storage medium.

According to a first aspect of embodiments of the present disclosure, there is provided an image processing method, including:

acquiring two groups of combined images of an acquisition object; the method comprises the steps that one group of combined images comprise a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprise a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any group of combined images have a mapping relation;

determining a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information characterizes a distance of the acquisition object from the first camera.

In some embodiments, the determining a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image includes:

carrying out depth region division on the first depth image and the second depth image to obtain a first depth sub-image of different depth regions and a second depth sub-image of different depth regions; wherein different depth regions characterize local regions on the acquisition object that are at different distances from the first camera;

according to the mapping relation, acquiring a first plane sub-image corresponding to the first depth sub-image of each depth area on the first plane image, and a second plane sub-image corresponding to the second depth sub-image of each depth area on the second plane image;

establishing a homography submatrix according to a group of the first plane subimages and the second plane subimages with the same depth area;

and determining a homography matrix between the first planar image and the second planar image according to homography submatrices corresponding to the combination of the plurality of groups of the first planar sub-images and the second planar sub-images.

In some embodiments, the method comprises:

performing feature point matching on the first planar image and the second planar image to obtain a matching combination between pixel points in the first planar image and pixel points in the second planar image;

establishing a homography submatrix according to a group of the first plane subimages and the second plane subimages with the same depth area, including:

selecting a target matching combination of the first plane sub-image and the second plane sub-image which belong to the same depth region from the matching combinations of the pixel points;

and calculating the homography submatrix based on the target matching combination.

In some embodiments, different ones of the homography sub-matrices are weighted differently; and the weight of the homography submatrix corresponding to the target depth area is greater than that of the homography submatrix corresponding to the depth area except the target depth area.

In some embodiments, the method comprises:

performing offline calibration on the first camera to obtain a first calibration result; the first calibration result is used for correcting the image acquired by the first camera to obtain the first depth image and the second depth image;

performing off-line calibration on the second camera to obtain a second calibration result; the second calibration result is used for correcting the image acquired by the second camera to obtain the first plane image and the second plane image;

and determining the mapping relation based on the first calibration result and the second calibration result.

In some embodiments, the method comprises:

and registering the first plane image and the second plane image according to the homography matrix.

According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus including:

an acquisition module configured to acquire two sets of combined images of an acquisition object; the method comprises the steps that one group of combined images comprise a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprise a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any group of combined images have a mapping relation;

a determining module configured to determine a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information characterizes a distance of the acquisition object from the first camera.

In some embodiments, the determining module is further configured to perform depth region division on the first depth image and the second depth image, and obtain a first depth sub-image of a different depth region and a second depth sub-image of a different depth region; wherein different depth regions characterize local regions on the acquisition object that are at different distances from the first camera; according to the mapping relation, acquiring a first plane sub-image corresponding to the first depth sub-image of each depth area on the first plane image, and a second plane sub-image corresponding to the second depth sub-image of each depth area on the second plane image; establishing a homography submatrix according to a group of the first plane subimages and the second plane subimages with the same depth area; and determining a homography matrix between the first planar image and the second planar image according to homography submatrices corresponding to the combination of the plurality of groups of the first planar sub-images and the second planar sub-images.

In some embodiments, the apparatus comprises:

the characteristic matching module is configured to perform characteristic point matching on the first plane image and the second plane image to obtain a matching combination between pixel points in the first plane image and pixel points in the second plane image;

the determining module is further configured to select a target matching combination of the first planar sub-image and the second planar sub-image belonging to the same depth region from the matching combinations of the pixel points; and calculating the homography submatrix based on the target matching combination.

In some embodiments, the apparatus comprises:

the first calibration module is configured to perform offline calibration on the first camera to obtain a first calibration result; the first calibration result is used for correcting the image acquired by the first camera to obtain the first depth image and the second depth image;

the second calibration module is configured to perform offline calibration on the second camera to obtain a second calibration result; the second calibration result is used for correcting the image acquired by the second camera to obtain the first plane image and the second plane image;

a mapping module configured to determine the mapping relationship based on the first calibration result and the second calibration result.

In some embodiments, the apparatus comprises:

and the registration module is configured to register the first planar image and the second planar image according to the homography matrix.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to perform the image processing method as described in the first aspect above.

According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium including:

the instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image processing method as described in the first aspect above.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

in the embodiment of the disclosure, images acquired by combining the depth camera (first camera) and the plane camera (second camera) are utilized, so that when homography estimation between plane images is carried out, depth information in the depth images can be fused, and accuracy of homography estimation is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a flowchart illustrating an image processing method according to an embodiment of the disclosure.

FIG. 2 is an exemplary diagram of an application based on a homography matrix in an embodiment of the present disclosure.

Fig. 3 is a diagram illustrating an example of depth region division according to an embodiment of the disclosure.

Fig. 4 is an exemplary diagram of obtaining a calibration result in the embodiment of the present disclosure.

Fig. 5 is an exemplary diagram of performing image correction based on a calibration result in an embodiment of the present disclosure.

Fig. 6 is a schematic diagram of an image processing method in an embodiment of the disclosure.

Fig. 7 is an exemplary diagram of acquiring a planar sub-image in an embodiment of the present disclosure.

Fig. 8 is a diagram illustrating an image processing apparatus according to an exemplary embodiment.

FIG. 9 is a block diagram illustrating an electronic device apparatus in accordance with an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Fig. 1 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure, and as shown in fig. 1, the image processing method applied to an electronic device includes the following steps:

s11, acquiring two groups of combined images of an acquisition object; the method comprises the steps that one group of combined images comprise a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprise a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any group of combined images have a mapping relation;

s12, determining a homography matrix between the first planar image and the second planar image based on the mapping relation and the depth information included in the first depth image and the second depth image; wherein the depth information characterizes a distance of the acquisition object from the first camera.

In the embodiments of the present disclosure, the electronic device may be any server or terminal with data processing capability, and the terminal includes, but is not limited to, a mobile phone, a tablet Computer, or a Personal Computer (PC).

In step S11, the electronic device acquires two sets of combined images of a captured object, where the captured object may be a person, an animal, a plant, or the like, or may be a natural scene such as a blue sky, a white cloud, or a sunset. The two groups of combined images of the object are collected, and may be images shot corresponding to a short focus or images shot corresponding to a long focus, which is not limited in this embodiment of the present disclosure.

In an embodiment of the present disclosure, any set of combined images includes a depth image captured by a depth camera (first camera) and a planar image captured by a normal color camera (second camera). The depth image comprises depth information used for reflecting the distance between each part on the collected object and the first camera.

In the embodiment of the present disclosure, the intrinsic parameters of the first camera and the second camera for acquiring any group of combined images may be the same, and the extrinsic parameters of the first camera and the second camera may also be the same, where the intrinsic parameters are parameters related to the characteristics of the cameras, such as the focal length, pixel size, etc. of the cameras; extrinsic parameters are parameters in the world coordinate system such as the position of the camera, the direction of rotation, etc. Of course, the internal parameters of the first camera and the second camera for acquiring any set of combined images may be different, and the external parameters may also be different. The present disclosure may determine a mapping relationship between the depth image and the planar image in any set of combined images based on extrinsic parameters of the first camera and the second camera.

In step S12, the electronic device may determine a homography matrix between the first planar image and the second planar image based on the mapping relationship between the depth image and the planar image and the depth information included in the first depth image and the second depth image.

As mentioned above, the intrinsic parameters and the extrinsic parameters of the first camera and the second camera capturing any set of combined images can be the same, so that the two-dimensional plane images captured by the first camera and the second camera in the same set of combined images have the same information, and only the depth image includes the additional distance information. In this case, in the same combination, the mapping relationship between the depth image and the plane image is one-to-one mapping in rows and columns. In such an embodiment, when determining the homography matrix between the first planar image and the second planar image based on the mapping relationship and the depth information, for example, the average acquisition distances of the first camera in each group of combined images may be calculated based on the depth information of each pixel point included in the first depth image and the second depth image, respectively, and the difference between the average acquisition distances may be used for estimation of the subsequent homography matrix. Specifically, when the homography matrix is subsequently estimated, feature point detection and matching can be performed based on the values of the pixel points in the first plane image and the second plane image and the ratio between the average acquisition distances, and then the mapping relationship between the feature points is calculated, so that the homography matrix of the disclosure is obtained. And when the ratio between the average acquisition distances can be used as the position constraint condition of the matched pixel points during feature point detection and matching. For example, when the collection directions of the first planar image and the second planar image are consistent, the ratio between the average collection distances determined based on the first depth image and the second depth image is 2, and then for a certain pixel point a on the first planar image, a matched pixel point can be found in the pixel range of 2 times of the pixel point a in the second planar image, and therefore the homography matrix is further calculated based on the matched pixel point.

In this embodiment, based on the constraints of the depth information in the first depth image and the second depth image, the accuracy of feature point detection and matching can be improved, and thus the accuracy of the unicity matrix estimation can be improved.

In another embodiment, considering that the distances between different regions on the acquisition object and the first camera are different, the method may also perform region division based on different depth information on the depth image, and perform block-based unitary matrix estimation on the first planar image and the second planar image by combining with a mapping relationship, so as to improve the accuracy of the homography matrix through more detailed division.

It can be understood that, in the embodiment of the present disclosure, the images acquired by using the depth camera (first camera) and the plane camera (second camera) in combination enable the depth information in the depth images to be fused when performing homography estimation between the plane images, thereby improving the accuracy of the homography estimation.

In one embodiment, the method comprises:

After the homography matrix is determined, the homography matrix can be used for matching and overlapping the first plane image and the second plane image, for example, multi-pose face registration is carried out. Of course, the embodiments of the present disclosure are not limited to the registration by using the homography matrix, and may also use the homography matrix for image stitching, etc.

Fig. 2 is an exemplary diagram of an application based on a homography matrix in an embodiment of the present disclosure, and as shown in fig. 2, after an electronic device obtains two sets of images with different capture poses through a depth camera (i.e., a first camera) and an RGB camera (i.e., a second camera), the electronic device performs multi-pose registration on a color image 1 (i.e., a first plane image) and a color image 2 (i.e., a second plane image) in combination with a depth image 1 (i.e., a first depth image) and a depth image 2 (i.e., a second depth image).

As described above, after performing region division based on different depth information on the depth image, the first planar image and the second planar image may be subjected to block unitary matrix estimation by combining with the mapping relationship. In this embodiment, step S12 may specifically include the following steps:

carrying out depth region division on the first depth image and the second depth image to obtain a first depth sub-image of a different depth region and a second depth sub-image of the different depth region; wherein different depth regions characterize local regions on the acquisition object that are at different distances from the first camera;

Fig. 3 is a depth region division example diagram illustrating an embodiment of the present disclosure, and as shown in fig. 3, when depth regions are divided between a first depth image and a second depth image, a pixel point with depth information less than 50 centimeters (cm) may be divided into one depth region, a pixel point with depth information in a range of 50 centimeters to 100 centimeters may be divided into one depth region, a pixel point with depth information in a range of 100 centimeters to 200 centimeters may be divided into one depth region, a pixel point with depth information in a range of 200 centimeters to 350 centimeters may be divided into one depth region, and a pixel point with depth information greater than 350 centimeters may be divided into another depth region. Of course, the depth region division of the present disclosure is not limited to the above-described manner, and the depth region division manner may be determined by combining the determination accuracy requirement of the unitary matrix and the calculation amount. Overall, the more depth regions are divided, the higher the accuracy, but at the same time the larger the amount of computation may be. It should be noted that, in the embodiment of the present disclosure, the depth regions of the first depth image and the second depth image are divided in the same manner.

According to the method and the device, after the first depth sub-image of the different depth areas is obtained based on the first depth image and the second depth sub-image of the different depth areas is obtained based on the second depth image, the first plane sub-image corresponding to the first depth sub-image on the first plane image and the second plane sub-image corresponding to the second depth sub-image on the second plane image can be obtained according to the mapping relation. It should be noted that, in the embodiment of the present disclosure, the mapping relationship between the first depth image and the first planar image may be different from the mapping relationship between the second depth image and the second planar image. When the two groups of combined images have different mapping relationships, for example, the mapping relationships may include a first mapping relationship and a second mapping relationship, and a first plane sub-image corresponding to the first depth sub-image on the first plane image may be obtained according to the first mapping relationship, and a second plane sub-image corresponding to the second depth sub-image on the second plane image may be obtained according to the second mapping relationship.

It will be appreciated that since different depth sub-images include different depth information, the planar sub-images corresponding to different depth sub-images also correspond to different depth information. Subsequently, a homography submatrix, which is a mapping matrix representing local areas on different first planar images and second planar images acquired by the second camera, corresponding to the same depth area on the first depth image and the second depth image, may be established according to a set of first planar submatrix and second planar submatrix having the same depth area.

After the homography submatrices of the local areas on the first plane image and the second plane image are determined, the homography submatrices can be combined according to the area positions to obtain a homography matrix between the first plane image and the second plane image.

Exemplary, suppose H _i Is a homography submatrix of the ith block area, the homography between the first planar image and the second planar image can be expressed as the following formula (1):

wherein n represents the number of the block areas, and n is a positive integer greater than 1.

It should be noted that, in this embodiment, the capturing poses of the two sets of combined images may be different, but in different combinations, the distance from the capturing object by the first camera may be the same.

It can be understood that, according to the depth information included by the first depth image and the second depth image, the depth area is divided, then the first plane sub-image and the second plane sub-image of the same depth area are obtained according to the mapping relation, the homography sub-matrixes corresponding to the depth area are solved, and then the homography sub-matrixes are combined to obtain the homography matrixes corresponding to the first plane image and the second plane image, so that limitation caused by the fact that the homography matrixes are estimated only by means of plane image information is reduced, and accuracy of homography matrix estimation can be improved by means of area division mapping based on the depth information.

It should be noted that, in an embodiment of the present disclosure, when a homography submatrix is established based on a set of first and second planar sub-images with the same depth area, detection and matching of feature points may be directly performed on the first and second planar sub-images, so as to calculate a homography submatrix between the first and second planar sub-images based on matched feature point pairs.

In another embodiment of the present disclosure, the homography matrix between the first planar sub-image and the second planar sub-image can be calculated in other manners, and the method includes:

the establishing of the homography submatrix according to the group of the first plane subimage and the second plane subimage with the same depth area comprises:

In the embodiment of the disclosure, feature point matching can be performed based on a Scale Invariant Feature Transform (SIFT) algorithm, and Feature point detection matching can also be performed based on an accelerated Robust Feature (speedup Robust Features, SURF) and other algorithms.

In this embodiment, feature point matching may be performed on the first planar image and the second planar image to obtain a matching combination between the pixel points in the first planar image and the pixel points in the second planar image, and then a target matching combination belonging to the first planar sub-image and the second planar sub-image is selected and output from the matching combination of the pixel points according to the region information of the first planar sub-image and the second planar sub-image. For example, feature points in the matching combination are blocked, a mask mode is adopted, the areas of the first plane sub-image and the second plane sub-image are set to be 1, and the rest are set to be 0, so that a target matching combination is filtered out from the matching combination of the pixel points. Assuming that the matching combination of the first plane image and the second plane image is Feature1 and Feature2, the filtered target matching combination is Feature1_ Roi, feature2_ Roi, where Feature _ Roi is a subset of Feature to which a region limit is added with respect to Feature.

In one embodiment, different ones of the homography sub-matrices are weighted differently; and the weight of the homography submatrix corresponding to the target depth area is greater than that of the homography submatrix corresponding to the depth area except the target depth area.

In the embodiment of the disclosure, the weight of the homography submatrix corresponding to the target depth region is set to be larger, so as to improve the importance of the homography submatrix of the target depth region in the whole homography submatrix, and thus, when the homography submatrix is subsequently subjected to operations such as image matching or splicing, a better matching effect or splicing effect is obtained.

For example, when a portrait is acquired, the preferred acquisition distance is 100 cm to 300 cm, so that a depth area of 100 cm to 300 cm can be determined as a focus of attention, i.e., a target depth area of the present disclosure, and other background areas belong to non-focus areas; when a close shot is acquired, the distance of interest is typically less than 50 centimeters, and thus a depth zone of less than 50 centimeters may also be determined as the target depth zone. According to the method and the device, the target depth area can be set according to the acquisition mode of the acquisition object, and the weight corresponding to the homography submatrix of the target depth area is relatively large.

Book of JapaneseIn the open embodiment, it is assumed that the weight corresponding to the homography submatrix is W = (W) ₁ … w _n ) Then, the weighted homography matrix can be expressed as shown in the following equation (2):

H _result ＝W·H (2)

where H is the matrix shown in the foregoing formula (1).

In one embodiment, the method comprises:

calibrating the first camera to obtain a first calibration result; the first calibration result is used for correcting the image acquired by the first camera to obtain the first depth image and the second depth image;

calibrating the second camera to obtain a second calibration result; the second calibration result is used for correcting the image acquired by the second camera to obtain the first plane image and the second plane image;

When the images are collected through the cameras, the positions of pixel points in the depth images and the positions of the pixel points in the plane images can be different due to the pose difference of the first camera and the second camera in the same group of combined images, the original appearance of the collected object cannot be reflected really due to the deformation of the shot images caused by the possible distortion of the lens of the cameras, and the like, so that the camera imaging geometric model needs to be established through camera calibration to reduce the influence of the phenomena on the homography matrix estimation. The process of calibrating the camera is also the process of obtaining the internal parameters, the external parameters and the distortion parameters of the camera.

In the embodiment of the disclosure, a first calibration result of the first camera and a second calibration result of the second camera may be obtained by using an offline calibration method. The off-line calibration method is, for example, to provide the correspondence between an image point and its corresponding three-dimensional space point by using a three-dimensional calibration object with non-coplanar dedicated calibration markers and to calculate calibration parameters. Of course, in the case of missing calibration objects, the present disclosure may also adopt an online calibration manner, for example, a self-calibration manner based on scene constraints and a self-calibration manner based on geometric constraints.

Fig. 4 is an exemplary diagram of obtaining a calibration result in the embodiment of the present disclosure, and as shown in fig. 4, the depth image and the RGB image are calibrated offline to obtain the calibration result, where the calibration result includes an internal parameter, an external parameter, and a distortion parameter of a camera.

The present disclosure may perform offline calibration on a first camera and a second camera by using a checkerboard, so as to obtain calibration results (including a first calibration result and a second calibration result) of the cameras (including the first camera and the second camera). Based on the internal parameters and distortion parameters in the calibration result, the depth image acquired by the first camera can be corrected, and the plane image acquired by the second camera can be corrected. In addition, based on the extrinsic parameters, a mapping relationship between a depth image acquired by the first camera and a plane image acquired by the second camera can be established, and if the relative positions of the first camera and the second camera are different in pose when two groups of combined images are acquired, the first mapping relationship and the second mapping relationship need to be established respectively.

Fig. 5 is an exemplary diagram of performing image correction based on the calibration result in the embodiment of the present disclosure, as shown in fig. 5, a depth camera is a first camera of the present disclosure, an RGB camera is a second camera of the present disclosure, a depth image obtained by the depth camera may be stereoscopically corrected by using the calibration result, so as to obtain a corrected depth image (including a first depth image and a second depth image), and an RGB image obtained by the RGB camera may be stereoscopically corrected by using the calibration result, so as to obtain a corrected RGB image (including a first plane image and a second plane image). In addition, the mapping relationship between the corrected depth image and the plane image is established by combining the external parameters in the calibration result.

According to the method, after the camera is calibrated, the acquired images are corrected by using the calibration result, and the homography matrix is estimated based on the two corrected combined images, so that error mapping caused by the phenomena of deformation and the like of the uncorrected images is reduced, and the accuracy of homography matrix estimation can be improved.

Fig. 6 is a schematic diagram of an image processing method according to an embodiment of the disclosure, and as shown in fig. 6, a depth map 1 (a first depth image) and a plane image 1 (a first plane image) are taken as an example to describe a previous processing process, where the depth map 1 is an image obtained by performing image correction on an image acquired by a depth camera (a first camera), the plane image 1 is an image obtained by performing image correction on an image acquired by an RGB camera (a second camera), and an image correction method may be as shown in fig. 4 and fig. 5. After the corrected first depth image is obtained, depth area division can be performed, and meanwhile, a first plane sub-image (plane sub-image 1) corresponding to the first depth sub-image on the first plane image after the depth area division is obtained on the basis of the mapping relation between the depth image and the plane image. Similarly, the same operation is performed on the depth map 2 (second depth image) and the planar image 2 (second planar image), and a second planar sub-image (planar sub-image 2) corresponding to the second depth sub-image obtained by dividing the depth region on the second planar image is obtained. Fig. 7 is an exemplary diagram for obtaining a planar sub-image in an embodiment of the present disclosure, and as shown in fig. 7, after depth regions of a depth image are divided, different depth regions in the depth image may be obtained, and further, remapping is performed based on a correspondence of stereo correction, so as to obtain different regions in the planar image, that is, a planar sub-image is obtained. The correspondence of stereo correction, i.e. the aforementioned mapping relationship, is described. As shown in fig. 6, feature point detection and matching are also performed on the planar image 1 and the planar image 2 to obtain a feature point set 1 and a feature point set 2, each feature point in the feature point set 1 and a feature point in the feature point set 2 have a one-to-one correspondence relationship, and the feature point set 1 and the feature point set 2 are matching combinations between pixel points in the first planar image and pixel points in the second planar image. Subsequently, feature point blocking is performed in a mask mode based on the planar sub-image and the feature point set, so as to obtain a region feature point subset 1 belonging to the planar sub-image 1 and a region feature point subset 2 belonging to the planar sub-image 2, where the region feature point subset 1 and the region feature point subset 2 are the target matching combination mentioned in the present disclosure. Subsequently, the list corresponding to the depth region can be calculated according to the region feature point subset 1 and the region feature point subset 2Stress submatrix H _i . And combining the homography submatrices according to the region positions to obtain a homography matrix between the plane image 1 and the plane image 2.

Fig. 8 is a diagram illustrating an image processing apparatus according to an exemplary embodiment. Referring to fig. 8, the apparatus includes:

an acquisition module 101 configured to acquire two sets of combined images of an acquisition object; the method comprises the steps that one group of combined images comprise a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprise a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any group of combined images have a mapping relation;

a determining module 102 configured to determine a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information characterizes a distance of the acquisition object from the first camera.

In some embodiments, the determining module 102 is further configured to perform depth region division on the first depth image and the second depth image, to obtain a first depth sub-image of a different depth region and a second depth sub-image of a different depth region; wherein different depth regions represent local regions on the acquisition object that are at different distances from the first camera; according to the mapping relation, a first plane sub-image corresponding to the first depth sub-image of each depth area on the first plane image and a second plane sub-image corresponding to the second depth sub-image of each depth area on the second plane image are obtained; establishing a homography submatrix according to a group of the first plane subimages and the second plane subimages with the same depth area; and determining a homography matrix between the first planar image and the second planar image according to homography submatrices corresponding to the combination of the plurality of groups of the first planar sub-images and the second planar sub-images.

In some embodiments, the apparatus comprises:

a feature matching module 103, configured to perform feature point matching on the first planar image and the second planar image, so as to obtain a matching combination between a pixel point in the first planar image and a pixel point in the second planar image;

the determining module 102 is further configured to select a target matching combination of the first planar sub-image and the second planar sub-image belonging to the same depth region from the matching combinations of the pixel points; and calculating the homography submatrix based on the target matching combination.

In some embodiments, the apparatus comprises:

a first calibration module 104 configured to perform offline calibration on the first camera to obtain a first calibration result; the first calibration result is used for correcting the image acquired by the first camera to obtain the first depth image and the second depth image;

a second calibration module 105, configured to perform offline calibration on the second camera to obtain a second calibration result; the second calibration result is used for correcting the image acquired by the second camera to obtain the first plane image and the second plane image;

a mapping module 106 configured to determine the mapping relationship based on the first calibration result and the second calibration result.

In some embodiments, the apparatus comprises:

a registration module 107 configured to register the first planar image and the second planar image according to the homography matrix.

With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

Fig. 9 is a block diagram illustrating an electronic device apparatus 800 in accordance with an example embodiment. For example, the apparatus 800 may be a server or a terminal such as a mobile phone.

Referring to fig. 9, the apparatus 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operation at the device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

Power components 806 provide power to the various components of device 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 800.

The multimedia component 808 includes a screen that provides an output interface between the device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the device 800 is in an operational mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect the open/closed state of the device 800, the relative positioning of components, such as a display and keypad of the apparatus 800, the sensor assembly 814 may also detect a change in position of the apparatus 800 or a component of the apparatus 800, the presence or absence of user contact with the apparatus 800, orientation or acceleration/deceleration of the apparatus 800, and a change in temperature of the apparatus 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communications between the apparatus 800 and other devices in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as Wi-Fi,4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

A non-transitory computer readable storage medium in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform a method of image processing, the method comprising:

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that the method comprises:

2. The method of claim 1, wherein the determining a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image comprises:

3. The method of claim 2, wherein the method comprises:

matching feature points of the first plane image and the second plane image to obtain a matching combination between pixel points in the first plane image and pixel points in the second plane image;

4. The method of claim 2, wherein different homography submatrices in the homography matrix are weighted differently; and the weight of the homography submatrix corresponding to the target depth area is greater than that of the homography submatrix corresponding to the depth area except the target depth area.

5. The method according to claim 1, characterized in that it comprises:

6. The method according to claim 1, characterized in that it comprises:

7. An image processing apparatus, characterized in that the apparatus comprises:

8. The apparatus of claim 7,

the determining module is further configured to perform depth region division on the first depth image and the second depth image to obtain a first depth sub-image of a different depth region and a second depth sub-image of the different depth region; wherein different depth regions represent local regions on the acquisition object that are at different distances from the first camera; according to the mapping relation, acquiring a first plane sub-image corresponding to the first depth sub-image of each depth area on the first plane image, and a second plane sub-image corresponding to the second depth sub-image of each depth area on the second plane image; establishing a homography submatrix according to a group of the first plane subimages and the second plane subimages with the same depth area; and determining a homography matrix between the first planar image and the second planar image according to homography submatrices corresponding to the combination of the plurality of groups of the first planar sub-images and the second planar sub-images.

9. The apparatus of claim 8, wherein the apparatus comprises:

10. The apparatus of claim 8, wherein different homography sub-matrices in the homography matrix are weighted differently; and the weight of the homography submatrix corresponding to the target depth area is greater than that of the homography submatrix corresponding to the depth area except the target depth area.

11. The apparatus of claim 7, wherein the apparatus comprises:

and the mapping module is configured to determine the mapping relation based on the first calibration result and the second calibration result.

12. The apparatus of claim 7, wherein the apparatus comprises:

13. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to perform the image processing method of any one of claims 1 to 6.

14. A non-transitory computer readable storage medium, instructions in which, when executed by a processor of an electronic device, enable the electronic device to perform the image processing method of any one of claims 1 to 6.