WO2022218161A1

WO2022218161A1 - Method and apparatus for target matching, device, and storage medium

Info

Publication number: WO2022218161A1
Application number: PCT/CN2022/084323
Authority: WO
Inventors: 李泉录; 李若岱; 马堃
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-04-16
Filing date: 2022-03-31
Publication date: 2022-10-20
Also published as: CN113159161A; TW202242716A

Abstract

The present application provides a method and apparatus for target matching, a device, and a storage medium. The target matching method comprises: acquiring a first image and a second image, wherein the first image and the second image are obtained through photography by different photographing components on a target to be matched; obtaining a parallax of the target between the first image and the second image on the basis of a first distance between the target and at least one photographing component; and according to the parallax, determining a matching feature point pair with respect to the target in the first image and the second image. According to the solution above, the accuracy of target matching between different images can be improved.

Description

Method, apparatus, device and storage medium for target matching

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to the Chinese patent application with application number 202110413815.2 filed on April 16, 2021, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the technical field of image processing, and in particular, to a method, apparatus, device and storage medium for target matching.

Background technique

Object matching in images is a fundamental problem in computer vision, and the accuracy of object matching will affect the operations after object matching. The common target matching method in images mainly uses the two closest feature points in the two images to be matched as the matching point pair. Taking face matching as an example, it is mainly to judge the offset of the faces in the two images. If the coordinates of the faces in the vertical direction in the two images are the same, and the difference in the horizontal direction is the smallest, it is considered to be a mutual match. face. The problem with this method is that in the case of multiple faces in the image, the face with the smallest position difference in the two images may not be the face that actually matches each other.

SUMMARY OF THE INVENTION

The present application provides at least one method, apparatus, device and storage medium for target matching.

The present application provides a target matching method, including: acquiring a first image and a second image, wherein the first image and the second image are obtained by shooting a target to be matched by different shooting components; based on the relationship between the target and at least one shooting component The first distance between the first image and the second image is obtained to obtain the parallax of the target between the first image and the second image; according to the parallax, the matching feature point pair about the target in the first image and the second image is determined.

By obtaining the parallax of the target between the first image and the second image based on the first distance between the target and the at least one photographing component, and determining the matching feature point pair of the target according to the parallax, the first image and the second image can be effectively realized. Object matching between two images. In addition, compared with the method of directly using the feature point with the closest position in the two images as the matching feature point pair, the present application obtains a more accurate parallax by referring to the distance between the target and the shooting component, and then uses the parallax to obtain Matching point pairs improves the accuracy of target matching between different images.

Wherein, determining the pair of matching feature points about the target in the first image and the second image according to the parallax includes: determining, based on the parallax, a reference position corresponding to the first feature point in the second image, wherein the first feature point is a pair of The first image is obtained by performing target feature point detection; based on the reference position, a second feature point matching the first feature point is selected in the second image, wherein the second feature point is the target feature point detection performed on the second image. owned.

By determining the reference position corresponding to the first feature point in the second image based on the parallax, the second feature point is determined according to the reference position, the area in the second image that matches the first feature point is reduced, and the match can be quickly found Point to improve the matching accuracy.

Wherein, based on the reference position, selecting a second feature point matching the first feature point in the second image includes: determining a candidate region including the reference position in the second image, and selecting a candidate region that matches the first feature point in the candidate region The second feature point whose positional relationship of the epipolar line corresponding to the point in the second image satisfies the preset requirement is taken as the second feature point matching the first feature point.

By combining the two dimensions of disparity and epipolar geometric constraints to determine the second feature point that matches the first feature point, the accuracy of target matching is improved compared to using a single dimension.

Wherein, the epipolar line corresponding to the first feature point in the second image is determined by the following operations: the epipolar line is obtained by using the coordinates of the first feature point and the fundamental matrix.

The preset requirement is that the distance between the second feature point and the epipolar line in the candidate region is the smallest.

According to the principle of epipolar geometric constraint, the second feature point corresponding to the first feature point should be on the epipolar line, but errors will inevitably occur during the target matching process, resulting in the second feature point not being on the epipolar line, so the candidate is selected. The feature point with the smallest distance from the epipolar line in the region is used as the second feature point corresponding to the first feature point, which can improve the success rate of target matching.

Wherein, the first image or the second image is the reference image, the photographing component corresponding to the reference image is the reference photographing component, and the first distance is the distance between the target and the reference photographing component; the first distance is obtained through the following steps: determining that the target is in the reference The first size of the corresponding target area in the image; the first distance is obtained by using the ratio between the first size and the preset size.

By using the ratio between the first size of the target corresponding to the target area in the reference image and the preset size to determine the first distance, the technical effect that the distance between the target and the shooting component can be determined by using a single image is achieved, which breaks the The traditional perception of the distance between the target and the photographed component can be obtained by combining at least two or more images.

Wherein, the first size includes the first width and the first height of the target area, and the preset size includes the preset width and the preset height; using the ratio between the first size and the preset size to obtain the first distance, including: obtaining a first ratio between the first width and the preset width, and a second ratio between the first height and the preset height; the second distance between the target and the reference shooting component is obtained based on the first ratio, and based on the second The ratio obtains the third distance between the target and the reference photographing component; the first distance is obtained based on the second distance and the third distance.

The first distance is obtained by the weighted summation of the second distance and the third distance, which can comprehensively consider the results in the width and height directions, rather than using the second or third distance obtained in a single direction as the first distance In other words, the accuracy of the first distance can be improved.

Wherein, obtaining the second distance between the target and the reference photographing component based on the first ratio includes: multiplying the first ratio by the first focal length of the reference photographing component in the width direction to obtain the second distance; obtaining the target based on the second ratio The third distance from the reference photographing component includes: multiplying the second ratio by the second focal length of the reference photographing component in the height direction to obtain the third distance; and obtaining the first distance based on the second distance and the third distance, The method includes: weighted summation of the second distance and the third distance to obtain the first distance.

Because the resolution of the reference capture assembly may be different in the width or height direction, by multiplying the first ratio by the first focal length of the reference capture assembly in the width direction, and multiplying the second ratio by the reference capture assembly in the height direction Multiplying the second focal length of , the second and third distances are more accurate.

Wherein, determining the first size of the target corresponding to the target area in the reference image includes any one of the following: obtaining the first size based on the coordinates of at least two first feature points obtained by detecting the target feature points on the reference image; or, based on The size of the area obtained by performing the target area detection on the reference image is obtained to obtain the first size.

By obtaining the first size of the target area according to the coordinates of the first feature point or according to the size of the area obtained by detecting the target area on the first image, the process is simple and the computing power required by the device is low.

Wherein, the first distance is the distance between the target and the reference photographing component, and the reference photographing component is the photographing component corresponding to the first image or the second image; The parallax between an image and the second image includes: obtaining the parallax based on the first distance, the focal length of the reference shooting component, and the baseline between the shooting components corresponding to the first image and the second image.

By obtaining the parallax of the target between the first image and the second image based on the first distance, the focal length of the reference shooting component, and the baseline between the shooting components corresponding to the first image and the second image, the whole process is simple and requires The computing power is low, so that the processing resources of the execution device can be saved.

The present application provides a target matching device, comprising: an image acquisition module for acquiring a first image and a second image, wherein the first image and the second image are obtained by shooting a target to be matched by different shooting components; parallax acquisition a module for obtaining the parallax of the target between the first image and the second image based on the first distance between the target and the at least one photographing component; the matching module for determining the difference between the first image and the second image according to the parallax Matching feature point pairs about the target.

The present application provides an electronic device, including a memory and a processor, where the processor is configured to execute program instructions stored in the memory, so as to implement the target matching method in the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the target matching method in the above-mentioned first aspect is implemented.

In the above scheme, by obtaining the parallax of the target between the first image and the second image based on the first distance between the target and at least one shooting component, and determining the matching feature point pair of the target according to the parallax, the first Object matching between the image and the second image. In addition, compared with the method of directly using the feature point with the closest position in the two images as the matching feature point pair, the present application obtains a more accurate parallax by referring to the distance between the target and the shooting component, and then uses the parallax to obtain Matching point pairs improves the accuracy of target matching between different images.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

The accompanying drawings herein illustrate embodiments consistent with the present application, and together with the description, serve to explain the technical solutions of the present application.

Fig. 1 is a schematic flow chart 1 of an embodiment of a target matching method of the present application;

FIG. 2 is a second schematic flowchart of an embodiment of a target matching method of the present application;

3 is a schematic structural diagram of an embodiment of a target matching device of the present application;

4 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed ways

The solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present application.

The term "and/or" in this paper is used to describe the relationship between related objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this document generally indicates that there is an "or" relationship between the related objects before and after. Also, "multiple" herein means two or more than two. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

The present application provides some object matching methods and apparatuses. Target matching refers to matching corresponding feature points belonging to the same target in the first image and the second image, so as to find out the same target in the first image and the second image. The target can be any object, such as a face, a building, a vehicle, and so on.

Please refer to FIG. 1 . FIG. 1 is a first schematic flowchart of an embodiment of a target matching method of the present application. Specifically, the target matching method may include the following steps:

Step S11: Acquire a first image and a second image, wherein the first image and the second image are obtained by photographing the target to be matched by different photographing components, respectively.

Wherein, the shooting components corresponding to the first image and the second image can be placed arbitrarily, as long as the imaging planes of the first image and the second image are parallel, for example, the two shooting components can be placed side by side in the horizontal direction, It can also be placed side by side in the vertical direction. Two shooting components can form a binocular camera system. In the embodiment of the present disclosure, two photographing components are placed side by side in the horizontal direction as an example. Among them, the device that performs target matching and the photographing component may be an integrated device, or may be independent devices. Integrated means that the device that performs target matching and the photographing component can be controlled by the same processor, and independent means that the execution Target matching equipment and shooting components are controlled by different processors. Wherein, the first image and the second image may be images that have not been image-processed, or may be images that have been image-processed. Image processing can be adjusting brightness, resolution, etc. Further, the modality of the first image and the second image may be the same or different. For example, the first image and the second image may be both visible light images or infrared light images, or one of them may be a visible light image and the other may be an infrared light image. image. The form of the first image and the second image is not specified here.

Both the first image and the second image include at least one target, and one of the targets is used as the target to be matched, so as to match the feature points of the target in the first image and the second image, and then compare the first image and the second image. Find the target in the image to achieve target matching. In this embodiment of the present disclosure, the target is a human face as an example.

Step S12: Obtain the parallax of the target between the first image and the second image based on the first distance between the target and the at least one photographing component.

Wherein, the first distance between the target and at least one photographing component may be the distance between the target and one of the photographing components, or the distance from the position between the two photographing components. Wherein, the distance from the position between the two photographing components may specifically be the distance from the midpoint of the baselines of the two photographing components. Because the first distance from the photographing components is different, the parallax between the images obtained by photographing is also different. Specifically, the longer the first distance between the target and the photographing component, the smaller the parallax. Similarly, the closer the first distance between the target and the photographing component is, the greater the parallax. Therefore, the magnitude of the parallax can be determined according to the first distance between the target and the photographing component.

Step S13: According to the parallax, determine the matching feature point pair about the target in the first image and the second image.

The method of determining the matching feature point pair about the target in the first image and the second image according to the parallax may be to first determine a certain feature point in one of the images, and then determine the corresponding feature point in the other image according to the parallax Matching points to obtain the matching feature point pair of the target. If the target is a human face, the feature points related to the target in the first image and the second image may be obtained by performing feature detection on a region containing a human face in the images.

In the above solution, by obtaining the parallax of the target between the first image and the second image based on the first distance between the target and the at least one shooting component, and determining the matching feature point pair of the target according to the parallax, the first step can be effectively achieved. Object matching between an image and a second image. In this way, compared to the method of directly using the closest feature point in the two images as the matching feature point pair, the present application obtains a more accurate parallax by referring to the distance between the target and the shooting component, and then utilizes the parallax The matching feature point pairs are obtained, which improves the accuracy of target matching between different images.

In some disclosed embodiments, the first image or the second image is a reference image, the photographing component corresponding to the reference image is the reference photographing component, and the first distance is the distance between the target and the reference photographing component. Wherein, the method of obtaining the first distance may be:

First, determine the first size of the target corresponding to the target area in the reference image. Optionally, the first size includes a first width and a first height of the target area. The target area may be an area corresponding to a specific part of the target in the reference image. For example, if the target is a human face, the target region may be a corresponding region in the reference image of a face region composed of eyebrows, eyes, nose and mouth. There are various ways to specifically determine the first size of the target corresponding to the target area in the reference image.

For example, the first method is to obtain the first size based on the coordinates of the first feature point obtained by performing the target feature point detection on the reference image. The embodiment of the present disclosure takes the reference image as the first image as an example. For example, by inputting the first image and the second image into the first target detection deep neural network, the feature point coordinate information (X11, Y11), . . . (Xln, Yln) of the target to be detected in the first image and the second image are obtained The feature point coordinate information about the target to be detected in (Xr1, Yr1),...(Xrn, Yrn). Wherein, the first width may be the difference between the maximum value and the minimum value in all the width coordinates of the reference image, expressed as the first width by a formula

w=max(Xl1,...Xln)-min(Xl1,...Xln)

The first height can be the difference between the maximum value and the minimum value among all the height coordinates of the reference image, expressed as the first height by a formula

h=max(Yl1,...Yln)-min(Yl1,...Yln)

When the feature point coordinates of the reference image are acquired, the feature point coordinate information of another image is simultaneously acquired, which simplifies the target matching process.

The second is to obtain the first size based on the size of the region obtained by performing target region detection on the reference image. The area obtained by performing the target area detection on the reference image may be the area included in the target detection frame, and the area included in the target detection frame may correspond to a specific part of the target. Optionally, the width of the target detection frame is used as the first width, and the height of the target detection frame is used as the first height. For example, taking the target as a face as an example, the target detection frame may be a face detection frame, wherein the area included in the face detection frame is an area jointly formed by eyebrows, eyes, nose and mouth. For example, the reference image is input into the second target detection deep neural network, and a face detection frame is output, wherein the width of the face detection frame can be used as the first width, and the height of the face detection frame can be used as the first height. By obtaining the first size of the target area according to the coordinates of the first feature point or according to the size of the area obtained by detecting the target area on the first image, the process is simple and the computing power required by the device is low.

Secondly, the first distance is obtained by using the ratio between the first size and the preset size. The preset size may include a preset width and a preset height. The ratio between the first size and the preset size may be the ratio between the widths of the two or the ratio between the heights, or a combination of the ratio between the widths and the height. The first distance may be obtained by performing perspective transformation using the first size. By using the ratio between the first size of the target corresponding to the target area in the reference image and the preset size to determine the first distance, the technical effect that the distance between the target and the shooting component can be determined by using a single image is achieved, which breaks the The traditional perception of the distance between the target and the photographed component can be obtained by combining at least two or more images.

Specifically, the first ratio between the first width and the preset width, and the second ratio between the first height and the preset height may be acquired. The second distance between the target and the reference photographing component is obtained based on the first ratio, and the third distance between the target and the reference photographing component is obtained based on the second ratio. Wherein, the preset size here is the real size of the preset target with the same attribute as the target. For example, when the target is a face, the preset size is the preset real size of other faces. Wherein, both the first size and the preset size may be sizes corresponding to the target preset part. For example, the preset size corresponds to the real size of an area composed of eyebrows, eyes, nose and mouth of the preset target, and the first size corresponds to a target composed of eyebrows, eyes, nose and mouth in the reference image size of the area.

The second distance may be obtained by multiplying the first ratio by the first focal length of the reference photographing component in the width direction. Specifically, the formula for obtaining the second distance D2 is:

D ₂ =W ₁ f _x /w

Wherein, W1 is the preset width; w is the _first width; _fx is the first focal length of the reference photographing component in the width direction, and different photographing components have different focal lengths in the width direction. Wherein, f _x can be obtained by calibration of two reference shooting components, mainly by converting the real focal length of the reference shooting components to obtain the first focal length in the width direction and the second focal length in the height direction. The definition of the width and height here is consistent with the definition of the width and height of the reference image.

The third distance may be obtained by multiplying the second ratio by the second focal length of the reference photographing component in the height direction. Specifically, the formula for obtaining the third distance D3 may be

D ₃ =H ₁ f _y /h

Wherein, H ₁ refers to the preset height; f _y refers to the second focal length of the reference shooting component in the height direction, and different shooting components have different focal lengths in the height direction; h refers to the first height.

Finally, based on the second distance and the third distance, the first distance is obtained. Specifically, the second distance and the third distance may be weighted and summed to obtain the first distance. where the formula for the weighted summation can be

D ₁ =λD ₂ +(1-λ)D ₃

Wherein, D ₁ refers to the first distance, D ₂ refers to the second distance, D ₃ refers to the third distance, and λ is a preset parameter, and 0≤λ≤1, for example, the value of λ is 0.5. In the embodiment of the present disclosure, λ is a fixed parameter, and the value of λ is mainly related to the resolution of the reference photographing component. If the resolution of the reference photographing component is higher, the value of λ is closer to 0.5. However, in other disclosed embodiments, the value of λ can also be dynamically adjusted. Generally, if the absolute value of the difference between the acquired second distance and the third distance is smaller than the first difference, λ is set to 0.5; if the absolute value of the difference between the acquired second distance and the third distance is greater than the first difference, the value of λ is greater than 0.5 or less than 0.5. Optionally, if the second distance is greater than the third distance, the value of λ is greater than 0.5, and if the second distance is less than the third distance, the value of λ is less than 0.5. And the larger the absolute value of the difference between the two is, the farther the λ value is from 0.5.

Further, since the resolution of the reference photographing component in the width or height direction may be different, by multiplying the first ratio by the first focal length of the reference photographing component in the width direction, and multiplying the second ratio by the reference photographing component in the The second focal length in the height direction is multiplied, and the obtained second and third distances are more accurate.

In some disclosed embodiments, the first distance is the distance between the target and the reference photographing component, and the reference photographing component may be the photographing component corresponding to the first image or the second image. Wherein, based on the first distance between the target and the reference shooting component, the disparity of the target between the first image and the second image is obtained, including:

The parallax is obtained based on the first distance, the focal length of the reference photographing component, and the baseline between the photographing components corresponding to the first image and the second image. Specifically, the formula for obtaining the parallax Δd is as follows:

Δd=f _x B/D ₁

Among them, Δd is the parallax, f _x is the focal length of the reference photographing component in the width direction, D ₁ is the first distance, and B is the length of the baseline. The length of the baseline can be obtained by calibrating the two shooting components. By obtaining the parallax of the target between the first image and the second image based on the first distance, the focal length of the reference shooting component, and the baseline between the shooting components corresponding to the first image and the second image, the whole process is simple and requires The computing power is low, so that the processing resources of the execution device can be saved.

Wherein, according to the parallax, the step of determining the matching feature point pair about the target in the first image and the second image includes: based on the parallax, determining a reference position corresponding to the first feature point in the second image. The first feature point is obtained by detecting target feature points on the first image. Assuming that the coordinates of the first feature point in the first image are (x _ln , y _ln ) and the parallax is Δd, the corresponding reference position of the first feature point in the second image should be (x _ln +Δd,y _ln + Δd). That is, the second feature point corresponding to the first feature point in the second image should be near the reference position (x _ln +Δd, y _ln +Δd). Then, based on the reference position, a second feature point that matches the first feature point is selected in the second image. The second feature point is obtained by performing target feature point detection on the second image. Wherein, the feature point closest to the reference position in the second image may be selected as the second feature point matching the first feature point. The distance here can be the Euclidean distance. By determining the reference position corresponding to the first feature point in the second image based on the parallax, the second feature point is determined according to the reference position, the area in the second image that matches the first feature point is reduced, and the match can be quickly found Point to improve the matching accuracy. On the other hand, because the area matching the first feature point in the second image is reduced, the matching speed is also greatly improved, and the power consumption of the apparatus for performing the target matching method is reduced.

In some disclosed embodiments, according to the parallax, the step of determining the matching feature point pair about the target in the first image and the second image further includes determining the epipolar line corresponding to the first feature point in the second image. Wherein, the coordinates of the first feature point and the basic matrix can be used to obtain the epipolar line corresponding to the first feature point in the second image. Among them, the basic matrix is obtained by calibrating two shooting components. Specifically, the process of calibrating the two photographing components can be performed before step S11. The formula to obtain the epipolar line l can be:

l=(x _ln ,y _ln ,1)*F

Among them, (x _ln , y _ln , 1) is the homogeneous coordinate of the first feature point, and F is the fundamental matrix.

Wherein, a candidate area including the reference position may be determined in the second image, and a second feature point whose positional relationship with the epipolar line meets a preset requirement may be selected in the candidate area. The candidate area may be an area of a preset candidate size centered on the reference position. The determination of the preset candidate size can be set according to experience, and no specific provisions are made here. By combining the two dimensions of disparity and epipolar geometric constraints to determine the second feature point that matches the first feature point, the accuracy of target matching is improved compared to using a single dimension. Moreover, by selecting the feature point whose positional relationship with the epipolar line meets the preset requirements as the second feature point matching the first feature point, it is equivalent to reducing the feature matching area from two-dimensional to one-dimensional, reducing two The effect of placement between photographic components on target matching. And by performing target matching based on the two dimensions of disparity and epipolar geometric constraints, even when there are multiple faces in the first image and the second image, false matching is less likely to occur.

The preset requirement may be that the distance between the second feature point and the epipolar line in the candidate region is the smallest. That is, in the candidate region, the feature point with the smallest distance from the epipolar line is selected as the second feature point matching the first feature point. The distance here can be the Euclidean distance. According to the principle of epipolar geometric constraint, the second feature point corresponding to the first feature point should be on the epipolar line, but errors will inevitably occur during the target matching process, resulting in the second feature point not being on the epipolar line, so the candidate is selected. The feature point with the smallest distance from the epipolar line in the region is used as the second feature point corresponding to the first feature point, which can improve the success rate of target matching.

In the above solution, even if there are multiple interfering targets in the first image and the second image, by obtaining the first distance between the target and the shooting component and obtaining the parallax, the feature point pair belonging to the target is determined according to the parallax, The situation of incorrectly matching the feature points of the target with the feature points of other interfering targets is reduced, thereby improving the extraction accuracy of the matching feature points of the target when there are many interfering targets.

To better understand the technical solutions proposed by the embodiments of the present disclosure, please refer to the following examples.

Referring to FIG. 2 at the same time, FIG. 2 is a second schematic flowchart of an embodiment of a target matching method of the present application. As shown in FIG. 2 , the target matching method proposed by the embodiment of the present disclosure can be subdivided into the following steps:

Step S11: Acquire a first image and a second image, wherein the first image and the second image are obtained by photographing the target by different photographing components respectively.

For the specific manner of acquiring the first image and the second image, reference may be made to the above, and details are not described herein again.

Step S121: Obtain the first distance between the target and the reference photographing component.

The reference photographing component may be a photographing component corresponding to the first image, or may be a photographing component corresponding to the second image. For a specific manner of obtaining the first distance between the target and the reference photographing component, reference may be made to the above, and details are not described herein again.

Step S122: Based on the first distance, obtain the parallax of the target between the first image and the second image.

The method of obtaining the parallax can refer to the above, and details are not repeated here.

Step S131 : Determine a reference position corresponding to the first feature point in the second image based on the parallax, wherein the first feature point is obtained by performing target feature point detection on the first image.

The manner of determining the reference position corresponding to the first feature point in the second image based on the parallax can be as described above, and details are not described herein again.

Step S132: Determine the epipolar line corresponding to the first feature point in the second image.

Specifically, the manner of determining the epipolar line corresponding to the first feature point in the second image is as described above, and details are not repeated here. In the example given in the embodiment of the present disclosure, step S131 and step S132 may be performed synchronously. Of course, in other embodiments, one may be performed first, and then the other may be performed. The execution order of step S131 and step S132 is not specified here.

Step S133: Determine a candidate region containing the reference position in the second image, and in the candidate region, select a second feature point whose positional relationship with the epipolar line meets the preset requirements, as the second feature point matching the first feature point. Feature points.

The preset requirements may be as described above; in the candidate area, the process of selecting the second feature point matching the first feature point is as described above, which will not be repeated here.

In the above solution, the second feature point matching the first feature point is determined by integrating the two dimensions of disparity and epipolar geometric constraint, which improves the accuracy of target matching compared to using a single dimension. Moreover, by selecting the feature point whose positional relationship with the epipolar line meets the preset requirements as the second feature point matching the first feature point, it is equivalent to reducing the feature matching area from two-dimensional to one-dimensional, reducing two The effect of placement between photographic components on target matching. And by performing target matching based on the two dimensions of disparity and epipolar geometric constraints, even when there are multiple faces in the first image and the second image, false matching is less likely to occur.

After the matching feature point pair about the target is obtained through the solution proposed in the embodiment of the present disclosure, the extracted matching feature point pair about the target can be further utilized, for example, depth acquisition, living body detection, body temperature measurement, etc. For example, in a multi-person temperature measurement scenario, it is necessary to locate the same face captured by different cameras in the binocular camera system, and then obtain the position of the face in the thermal imager according to the face coordinates, and finally perform temperature extraction. Therefore, face matching is a pre-task for rapid body temperature measurement in a multi-person scenario. With the solution provided by the embodiments of the present disclosure, a face matching result with high precision and high robustness can be obtained to assist the body temperature in a multi-person scenario. Measurement.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

The execution subject of the target matching method may be a target matching apparatus. For example, the target matching method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment with requirements such as visual positioning, three-dimensional reconstruction, and image registration. (User Equipment, UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices and autonomous vehicles, there are Robots with positioning and mapping requirements, medical imaging systems with registration requirements, glasses, helmets and other products for augmented reality or virtual reality. In some possible implementations, the target matching method may be implemented by the processor invoking computer-readable instructions stored in the memory.

Please refer to FIG. 3 , which is a schematic structural diagram of an embodiment of a target matching apparatus of the present application. The target matching device 30 includes an image acquisition module 31 , a parallax acquisition module 32 , and a matching module 33 . The image acquisition module 31 is used to acquire a first image and a second image, wherein the first image and the second image are obtained by shooting the target to be matched by different shooting components; the parallax acquisition module 32 is used for shooting based on the target and at least one The first distance between the components is used to obtain the parallax of the target between the first image and the second image; the matching module 33 is configured to determine the matching feature point pair about the target in the first image and the second image according to the parallax.

In the above scheme, by obtaining the parallax of the target between the first image and the second image based on the first distance between the target and at least one shooting component, and determining the matching feature point pair of the target according to the parallax, the first Object matching between the image and the second image. Moreover, compared with the method of directly using the feature points with the closest positions in the two images as the matching feature point pair, the present application obtains a more accurate parallax by referring to the distance between the target and the shooting component, and then utilizes the parallax The matching feature point pairs are obtained, which improves the accuracy of target matching between different images.

In some disclosed embodiments, the matching module 33 determines, according to the parallax, a pair of matching feature points about the target in the first image and the second image, including: determining a reference position corresponding to the first feature point in the second image based on the parallax, wherein , the first feature point is obtained by performing target feature point detection on the first image; based on the reference position, a second feature point matching the first feature point is selected in the second image, wherein the second feature point is the first feature point. The two images are obtained by detecting the target feature points.

In the above solution, by determining the reference position corresponding to the first feature point in the second image based on the parallax, the second feature point is determined according to the reference position, and the area in the second image that is matched with the first feature point is reduced, which can quickly Find matching point pairs to improve the matching accuracy.

In some disclosed embodiments, the matching module 33 selects a second feature point in the second image that matches the first feature point based on the reference position, including: determining a candidate region including the reference position in the second image, and selecting a candidate region in the candidate region. , a second feature point whose positional relationship with the epipolar line corresponding to the first feature point in the second image satisfies a preset requirement is selected as the second feature point matching the first feature point.

In the above solution, the second feature point matching the first feature point is determined by integrating the two dimensions of disparity and epipolar geometric constraint, which improves the accuracy of target matching compared to using a single dimension.

In some disclosed embodiments, the matching module 33 determines the epipolar line corresponding to the first feature point in the second image, including: obtaining the epipolar line by using the coordinates of the first feature point and the fundamental matrix.

In some disclosed embodiments, the preset requirement is that the distance between the second feature point and the epipolar line in the candidate region is the smallest.

In the above scheme, according to the principle of epipolar geometric constraint, the second feature point corresponding to the first feature point should be on the epipolar line, but errors inevitably occur during the target matching process, resulting in the second feature point not being on the epipolar line, Therefore, selecting the feature point with the smallest distance from the epipolar line in the candidate region as the second feature point corresponding to the first feature point can improve the success rate of target matching.

In some disclosed embodiments, the first image or the second image is a reference image, the photographing component corresponding to the reference image is the reference photographing component, and the first distance is the distance between the target and the reference photographing component; the parallax obtaining module 32 is further configured to execute The first distance is obtained by the following steps: determining the first size of the target corresponding to the target area in the reference image; and obtaining the first distance by using the ratio between the first size and the preset size.

In the above solution, by using the ratio between the first size of the target corresponding to the target area in the reference image and the preset size to determine the first distance, the technical effect that the distance between the target and the shooting component can be determined by using a single image is achieved. , which breaks the traditional cognition that it is necessary to combine at least two images to obtain the distance between the target and the shooting component.

In some disclosed embodiments, the first size includes a first width and a first height of the target area, and the preset size includes a preset width and a preset height; the parallax obtaining module 32 obtains by using the ratio between the first size and the preset size. The first distance includes: obtaining a first ratio between the first width and a preset width, and a second ratio between the first height and the preset height; and obtaining the first ratio between the target and the reference shooting component based on the first ratio. two distances, and the third distance between the target and the reference photographing component is obtained based on the second ratio; the first distance is obtained based on the second distance and the third distance.

In the above scheme, the first distance is obtained by weighting the second distance and the third distance, and the results in the width and height directions can be comprehensively considered. Compared with the second distance or the third distance obtained by using a single direction as For the first distance, the accuracy of the first distance can be improved.

In some disclosed embodiments, the parallax obtaining module 32 obtains the second distance between the target and the reference photographing component based on the first ratio, including: multiplying the first ratio by the first focal length of the reference photographing component in the width direction to obtain the second distance. ; Obtain the third distance between the target and the reference shooting component based on the second ratio, comprising: multiplying the second ratio by the second focal length of the reference shooting component in the height direction to obtain the third distance; Based on the second distance and the third distance Obtaining the first distance includes: weighted summation of the second distance and the third distance to obtain the first distance.

In the above solution, because the resolution of the reference photographing component in the width or height direction may be different, by multiplying the first ratio by the first focal length of the reference photographing component in the width direction, and multiplying the second ratio by the reference photographing component in the width direction. The second focal length in the height direction is multiplied, and the obtained second and third distances are more accurate.

In some disclosed embodiments, the parallax obtaining module 32 determines the first size of the target corresponding to the target area in the reference image, including: obtaining the first size based on the coordinates of at least two first feature points obtained by performing target feature point detection on the reference image. or, obtaining the first size based on the size of the region obtained by performing target region detection on the reference image.

In the above solution, the first size of the target area is obtained according to the coordinates of the first feature point or the size of the area obtained by detecting the target area on the first image, the process is simple, and the computing power required by the device is low.

In some disclosed embodiments, the first distance is the distance between the target and the reference photographing component, and the reference photographing component is the photographing component corresponding to the first image or the second image; the parallax obtaining module 32 is based on the distance between the target and at least one photographing component. The first distance is to obtain the parallax of the target between the first image and the second image, including: based on the first distance, the focal length of the reference shooting component, and the baseline between the shooting components corresponding to the first image and the second image, obtaining Parallax.

The above scheme obtains the parallax of the target between the first image and the second image based on the first distance, the focal length of the reference shooting component, and the baseline between the shooting components corresponding to the first image and the second image, and the whole process is simple , the required computing power is low, so that the processing resources of the execution device can be saved.

Please refer to FIG. 4 , which is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is configured to execute program instructions stored in the memory 41, so as to implement the steps in the above-mentioned embodiments of the target matching method. In a specific implementation scenario, the electronic device 40 may include, but is not limited to, a microcomputer and a server. In addition, the electronic device 40 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.

Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in the above-mentioned embodiment of the target matching method. The processor 42 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 42 may be an integrated circuit chip with signal processing capability. The processor 42 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 42 may be jointly implemented by an integrated circuit chip.

Please refer to FIG. 5 , which is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 50 stores program instructions 501 that can be executed by a processor, and the program instructions 501 are used to implement the steps in the above embodiments of the target matching method.

In the above scheme, by obtaining the parallax of the target between the first image and the second image based on the first distance between the target and the at least one shooting component, and determining the matching feature point pair of the target according to the parallax, the first image can be realized and the target match between the second image. In addition, compared with the method of directly using the feature point with the closest position in the two images as the matching feature point pair, the present application obtains a more accurate parallax by referring to the distance between the target and the shooting component, and then uses the parallax to obtain Matching point pairs improves the accuracy of target matching between different images.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, details are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other divisions. For example, modules or units may be combined or integrated. to another system, or some features can be ignored or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Claims

A target matching method, comprising:

acquiring a first image and a second image, wherein the first image and the second image are obtained by photographing the target to be matched by different photographing components;

obtaining the parallax of the target between the first image and the second image based on the first distance between the target and at least one of the photographing components;

According to the disparity, a matching feature point pair about the target in the first image and the second image is determined.
The method according to claim 1, wherein the determining, according to the parallax, a pair of matching feature points about the target in the first image and the second image, comprising:

determining, based on the parallax, a reference position corresponding to the first feature point in the second image, wherein the first feature point is obtained by performing target feature point detection on the first image;

Based on the reference position, a second feature point matching the first feature point is selected in the second image, wherein the second feature point is obtained by performing target feature point detection on the second image of.
The method according to claim 2, wherein the selecting a second feature point matching the first feature point in the second image based on the reference position comprises:

determining a candidate region in the second image that includes the reference location, and

In the candidate area, a second feature point whose positional relationship with the epipolar line corresponding to the first feature point in the second image satisfies a preset requirement is selected as a matching feature point with the first feature point. Second feature point.
The method according to claim 3, wherein the epipolar line corresponding to the first feature point in the second image is determined by the following operations:

Using the coordinates of the first feature point and the fundamental matrix, the epipolar line is obtained.
The method according to claim 3 or 4, wherein the preset requirement is that the distance between the second feature point and the epipolar line in the candidate region is the smallest.
The method according to any one of claims 1 to 5, wherein the first image or the second image is a reference image, the photographing component corresponding to the reference image is a reference photographing component, and the The first distance is the distance between the target and the reference shooting component; the first distance is obtained through the following steps:

determining the first size of the target corresponding to the target area in the reference image;

The first distance is obtained by using the ratio between the first size and the preset size.
The method according to claim 6, wherein the first size includes a first width and a first height of the target area, and the preset size includes a preset width and a preset height; The ratio between the first size and the preset size to obtain the first distance, including:

obtaining a first ratio between the first width and the preset width, and a second ratio between the first height and the preset height;

Obtaining a second distance between the target and the reference photographing component based on the first ratio, and obtaining a third distance between the target and the reference photographing component based on the second ratio;

Based on the second distance and the third distance, the first distance is obtained.
The method of claim 7, wherein:

The obtaining the second distance between the target and the reference photographing component based on the first ratio includes:

multiplying the first ratio by the first focal length of the reference photographing component in the width direction to obtain the second distance;

The obtaining the third distance between the target and the reference photographing component based on the second ratio includes:

multiplying the second ratio by the second focal length of the reference photographing component in the height direction to obtain a third distance;

The obtaining the first distance based on the second distance and the third distance includes:

The second distance and the third distance are weighted and summed to obtain the first distance.
The method according to any one of claims 6 to 8, wherein the determining the first size of the target corresponding to the target area in the reference image comprises any of the following:

obtaining the first size based on the coordinates of at least two first feature points obtained by performing target feature point detection on the reference image;

The first size is obtained based on a size of an area obtained by performing target area detection on the reference image.
The method according to any one of claims 1 to 9, wherein the first distance is a distance between the target and a reference photographing component, and the reference photographing component is the first image or the the photographing component corresponding to the second image; and obtaining the parallax of the target between the first image and the second image based on the first distance between the target and at least one of the photographing components, including :

The parallax is obtained based on the first distance, the focal length of the reference photographing component, and the baseline between the photographing components corresponding to the first image and the second image.
A target matching device, comprising:

an image acquisition module, configured to acquire a first image and a second image, wherein the first image and the second image are obtained by photographing the target to be matched by different photographing components;

a parallax obtaining module, configured to obtain the parallax of the target between the first image and the second image based on the first distance between the target and at least one of the shooting components;

A matching module, configured to determine, according to the parallax, a pair of matching feature points about the target in the first image and the second image.
An electronic device is characterized by comprising a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory, so as to implement the method of any one of claims 1 to 10.
A computer-readable storage medium having program instructions stored thereon, characterized in that, when the program instructions are executed by a processor, the method of any one of claims 1 to 10 is implemented.